Methods and Systems for Generating a Surface Data Projection that Accounts for Level of Detail

ABSTRACT

An exemplary virtual reality provider system generates a surface data projection based on surface data representative of color characteristics or depth characteristics of surfaces within a three-dimensional (“3D”) space. The surface data projection is generated for a portion of the 3D space. The virtual reality provider system applies an image transform operation to the surface data projection to transform the surface data projection. Specifically, the image transform operation transforms the surface data projection to account for a level of detail of the surfaces within the 3D space with respect to a particular vantage point within the 3D space. Corresponding methods and systems are also disclosed.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/421,654, filed May 24, 2019, and entitled “Methods and Systems forGenerating Virtual Reality Data that Accounts for Level of Detail,”which is a continuation of U.S. patent application Ser. No. 15/799,731,filed Oct. 31, 2017 and issued as U.S. Pat. No. 10,347,037, on Jul. 9,2019, and entitled “Methods and Systems for Generating and ProvidingVirtual Reality Data that Accounts for Level of Detail,” which is acontinuation-in-part application of U.S. patent application Ser. No.15/610,572, filed May 31, 2017 and issued as U.S. Pat. No. 10,311,630 onJun. 4, 2019 and entitled “Methods and Systems for Rendering Frames of aVirtual Scene from Different Vantage Points Based on a Virtual EntityDescription Frame of the Virtual Scene.” These applications are herebyincorporated by reference in their entirety.

BACKGROUND INFORMATION

Users may experience virtual three-dimensional (“3D”) spaces (e.g.,based on virtual, real-world, or mixed reality scenes) for variousreasons and in connection with various types of applications. Forexample, users may experience virtual 3D spaces for entertainmentpurposes, educational purposes, long-distance communication purposes,vicarious experience/travel purposes, or in connection with variousother purposes and/or applications.

Virtual reality is one example of an application where users experiencevirtual 3D spaces. Virtual reality media content may be used to immerseusers (i.e., viewers of the virtual reality media content) intointeractive virtual reality worlds that users may experience bydirecting their attention to any of a variety of things being presentedin the immersive virtual reality world at the same time. For example, atany time during the presentation of the virtual reality media content, auser experiencing the virtual reality media content may look around theimmersive virtual reality world in any direction, giving the user asense that he or she is actually present in and experiencing theimmersive virtual reality world from a particular viewpoint or vantagepoint within the immersive virtual reality world.

In some examples, users may desire the flexibility of being able toexperience a virtual 3D space (e.g., an immersive virtual reality world)from an arbitrary virtual vantage point within the virtual 3D space. Inother words, the user may wish to move around to different locationswithin the virtual 3D space at will to experience the virtual 3D space(e.g., to view objects presented within the virtual 3D space, etc.) fromarbitrary virtual vantage points anywhere within the virtual 3D spacethat the user may dynamically choose. To provide the user this freedomof movement to the different locations, conventional media playerdevices have typically received data representative of the virtual 3Dspace (e.g., 3D models of objects within the 3D space and the like)prior to the time when the user experiences the virtual 3D space. Forexample, a conventional media player device may download and store data(e.g., 3D models, textures, etc.) associated with a virtual 3D space ona local storage facility such as a hard drive of the media playerdevice, or may access data associated with the virtual 3D space from alocal physical medium accessible to the media player device (e.g., aphysical disc). Unfortunately, however, more setup may be required by auser prior to experiencing such a virtual 3D space (e.g., setupassociated with downloading, installing, loading into memory, etc., datacontent representative of the virtual 3D space), and it may be difficultfor the virtual 3D space in such examples to reflect live or real-timeupdates to the content due to the requirement for much of the data to bereceived by the media player device prior to the user experience.

Moreover, even if all the data representative of the virtual 3D spacewere to be transmitted to a media player device in real time, datarepresenting individual 3D models of objects included within the virtual3D space may allow the virtual 3D space to be rendered from arbitraryvirtual vantage points within the virtual 3D space, but may not bescalable to present larger virtual 3D spaces or virtual 3D spacesincluding more objects without sacrificing quality. For example, if thedata being transmitted to the media player device representsindividually renderable 3D models for each object included within thevirtual 3D space, a significant amount of additional data (e.g.,approximately twice as much data) may be needed to represent a virtual3D space with, for example, ten objects, as compared to the amount ofdata needed to represent a virtual 3D space with, for example, fiveobjects. Thus, even if data representative of a virtual 3D spaceincluding models of five objects can be transmitted to a media playerdevice in real time, this type of transmission may not be able to scaleto similarly represent ten objects or one hundred objects or more withinthe virtual 3D space.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate various embodiments and are a partof the specification. The illustrated embodiments are merely examplesand do not limit the scope of the disclosure. Throughout the drawings,identical or similar reference numbers designate identical or similarelements.

FIG. 1 illustrates an exemplary virtual scene capture system forrendering frames of a virtual scene from different vantage points basedon a virtual entity description frame of the virtual scene according toprinciples described herein.

FIG. 2 illustrates an exemplary configuration in which the virtual scenecapture system of FIG. 1 facilitates rendering frames of a virtual scenefrom different vantage points based on a virtual entity descriptionframe of the virtual scene according to principles described herein.

FIG. 3 illustrates an exemplary virtual scene including a plurality ofvirtual entities according to principles described herein.

FIG. 4 illustrates exemplary modifications that may be made to thevirtual scene of FIG. 3 according to principles described herein.

FIG. 5 illustrates exemplary virtual entity description frames that maybe generated by the virtual scene capture system of FIG. 1 according toprinciples described herein.

FIG. 6 illustrates exemplary three-dimensional (“3D”) rendering enginesthat render surface data frames representative of color and depth dataof surfaces of virtual objects visible from different vantage pointsbased on an exemplary virtual entity description frame according toprinciples described herein.

FIG. 7 illustrates a plurality of exemplary frame sequences of surfacedata frames representative of color and depth data of surfaces of anexemplary virtual object visible from different vantage points accordingto principles described herein.

FIG. 8 illustrates an exemplary configuration in which an exemplaryvirtual reality media content provider system generates virtual realitymedia content that is provided by way of a network to an exemplaryclient-side media player device used by a user to experience a virtualscene according to principles described herein.

FIG. 9 illustrates various exemplary types of media player devices thatmay be used by a user to experience virtual reality media contentaccording to principles described herein.

FIG. 10 illustrates an exemplary virtual reality experience in which auser is presented with exemplary virtual reality media contentrepresentative of a virtual scene as experienced from a dynamicallyselectable virtual vantage point corresponding to an exemplary arbitraryvirtual location with respect to the virtual scene according toprinciples described herein.

FIG. 11 illustrates an exemplary method for rendering frames of avirtual scene from different vantage points based on a virtual entitydescription frame of the virtual scene according to principles describedherein.

FIG. 12 illustrates an exemplary virtual reality provider system forgenerating and providing virtual reality data that accounts for level ofdetail according to principles described herein.

FIG. 13 illustrates an exemplary configuration in which an exemplaryimplementation of the virtual reality provider system of FIG. 12generates and provides virtual reality data accounting for level ofdetail to exemplary media player devices according to principlesdescribed herein.

FIG. 14 illustrates an exemplary virtual 3D space of a virtual scenebased upon which the virtual reality provider system of FIG. 12 maygenerate and provide virtual reality data that accounts for level ofdetail according to principles described herein.

FIGS. 15A through 15F illustrate exemplary pluralities of adjacentsurface data slices of the virtual 3D space of FIG. 14 according toprinciples described herein.

FIGS. 16A through 16C illustrate exemplary orthographic projectionsassociated with certain surface data slices selected from thepluralities of adjacent surface data slices illustrated in FIGS. 15Athrough 15F according to principles described herein.

FIG. 17 illustrates an exemplary slice atlas data structure formed by anarrangement of surface data slices in the pluralities of adjacentsurface data slices illustrated in FIGS. 15A through 15F according toprinciples described herein.

FIG. 18 illustrates a plurality of exemplary vantage points disposed atdifferent locations within the virtual 3D space of FIG. 14 according toprinciples described herein.

FIG. 19 illustrates another plurality of exemplary vantage pointsdisposed at different locations within the virtual 3D space of FIG. 14according to principles described herein.

FIGS. 20A through 20C illustrate exemplary image transform operationsthat may be performed on the surface data slices of the virtual 3D spacethat are included in the pluralities of adjacent surface data slicesillustrated in FIGS. 15A through 15F according to principles describedherein.

FIG. 21 illustrates how an exemplary instance of virtual reality datamay be generated to represent the virtual 3D space of FIG. 14 and toaccount for level of detail of surfaces included within the virtual 3Dspace of FIG. 14 with respect to a particular vantage point according toprinciples described herein.

FIG. 22 illustrates an exemplary method for generating and providingvirtual reality data that accounts for level of detail according toprinciples described herein.

FIG. 23 illustrates an exemplary computing device according toprinciples described herein.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Methods and systems for rendering frames of a virtual scene fromdifferent vantage points based on a virtual entity description frame ofthe virtual scene are described herein. For example, a virtual scenecapture system may maintain data representative of a plurality ofvirtual entities included within a virtual three-dimensional (“3D”)space of a virtual scene. The plurality of virtual entities may includeone or more virtual objects along with a plurality of virtual vantagepoints into the virtual 3D space (e.g., virtual locations, angles,viewpoints, etc., from which to view the one or more virtual objectsincluded within the virtual 3D space). In particular, the plurality ofvirtual vantage points may include at least a first virtual vantagepoint and a second virtual vantage point different from the firstvirtual vantage point.

Based on the maintained data representative of the plurality of virtualentities, the virtual scene capture system may generate a virtual entitydescription frame representative of a state of at least one virtualentity in the plurality of virtual entities at a particular point in atemporal sequence. For example, the virtual entity description frame maybe a key description frame that represents respective state informationfor all the virtual entities (i.e., virtual objects and virtual vantagepoints) included in the plurality of virtual entities, or may be anupdate description frame representing state information of only thosevirtual entities in the plurality of virtual entities that have changedsince a previous key description frame was generated.

Upon generating the virtual entity description frame, the virtual scenecapture system may provide the virtual entity description frame to aplurality of server-side 3D rendering engines associated with a contentprovider system to facilitate the 3D rendering engines in renderingframes of the virtual scene from different vantage points based on thevirtual entity description frame. For instance, the virtual scenecapture system may provide the virtual entity description frame to afirst 3D rendering engine that is associated with the first virtualvantage point and is configured to render, based on the virtual entitydescription frame, a first surface data frame representative of colorand depth data of surfaces of the virtual object visible from the firstvirtual vantage point at the particular point in the temporal sequence.Similarly, the virtual scene capture system may provide the virtualentity description frame to a second 3D rendering engine that isassociated with the second virtual vantage point and is configured torender, also based on the virtual entity description frame, a secondsurface data frame representative of color and depth data of surfaces ofthe virtual object visible from the second virtual vantage point at theparticular point in the temporal sequence.

The systems and methods for rendering frames of a virtual scene fromdifferent vantage points based on a virtual entity description frame ofthe virtual scene described herein may provide various advantages andbenefits. For example, the systems and methods described herein mayfacilitate users in experiencing virtual 3D spaces of virtual scenes. Asused herein, a “virtual 3D space” of a virtual scene may refer to arendering (e.g., a wholly virtualized rendering) of an environment or aworld (e.g., an immersive virtual reality world) that may be experiencedby a user in a similar way as the user might experience the real world.For example, a user experiencing the virtual scene may be able to moveabout within the virtual 3D space and look at and/or otherwise interactwith objects included within the virtual space. In some examples, avirtual 3D space may be wholly virtualized (e.g., computer generated)and rendered in a similar way as a real-world scene may be rendered. Inother examples, a virtual 3D space may be based, at least in part, onone or more real-world objects captured from a real-world scene.

In any case, the systems and methods described herein may facilitateusers in experiencing virtual 3D spaces of virtual scenes that arestreamed, in their entirety, from a provider system such that datarepresentative of the virtual 3D spaces and the virtual entitiesincluded therein do not need to be preloaded or stored on a media playerdevice prior to the experiencing of the virtual 3D space by the user ofthe media player device. For example, all the data needed for a mediaplayer device to present a virtual scene may be streamed to the mediaplayer device (e.g., in real time in certain implementations) so thatdata representative of virtual scene content does not need to bedownloaded, stored, or otherwise accessed (e.g., by way of a localphysical disc) prior to the presentation of the virtual scene to theuser.

Moreover, the systems and methods for rendering frames of a virtualscene from different vantage points based on a virtual entitydescription frame of the virtual scene described herein may facilitateproviding virtual reality media content representative of the virtualscene to media player devices in such a way that the virtual realitymedia content may be rendered from arbitrary virtual locations anddynamically selected virtual vantage points within the virtual 3D space.Specifically, as will be described in more detail below, by renderingframes of a virtual scene from different vantage points (e.g., thevirtual vantage points), the virtual scene capture system may includethe frames in a data pipeline configured to allow a media player deviceto render, in three dimensions, a virtual 3D space from arbitrary anddynamically selectable virtual vantage points based on a plurality oftwo-dimensional (“2D”) video streams included in the data pipeline andassociated with, for example, relatively fixed vantage points (e.g., thevirtual vantage points). As a result, the media player device may allowthe user to experience the virtual 3D space as if moving around freelywithin the virtual 3D space based on 2D video streams carrying datarepresentative of the virtual 3D space, rather than based on 3D modeldata representative of a variable and potentially unlimited number of 3Dmodels associated with the virtual 3D space. For example, rather thanproviding data representative of 3D models of every virtual objectincluded within the virtual 3D space, the data pipeline may provide 2Dvideo data (e.g., color data and depth data) representative of all thevirtual objects within the virtual 3D space from the virtual vantagepoints. As such, an unlimited number of objects may be represented in arendering of the virtual scene without the media player device having toreceive additional data or perform additional rendering work than wouldbe required for rendering the virtual scene with only one or twoobjects, for example.

Additionally, by maintaining and providing all the data representativeof the virtual scene to the media player devices without relying onpreloaded content already stored at the media player devices, the systemand methods described herein may allow virtual 3D spaces to be generatedor modified (e.g., in real time) by the provider without having tomodify preloaded data stored on the media player device. As a result,content creators responsible for generating a virtual scene or one ormore users experiencing the virtual scene may provide commands to thevirtual scene capture system to modify aspects of the virtual scene(e.g., to modify, replace, or remove virtual objects, etc.), and thesemodifications can be instantly reflected in the data being streamed tousers such that the virtual scene is modified in real time or near realtime.

Similarly, various operations that may be computationally expensive(e.g., prohibitively expensive for certain media player devices) may beperformed by powerful computing resources associated with the virtualscene capture system, which may be operated by a virtual reality mediaprovider and may be associated with much more powerful computingresources (e.g., large servers or the like) than, for example, the mediaplayer devices associated with users. For example, the virtual scenecapture system may perform computationally expensive physics operationswith respect to objects within a virtual scene, artificial intelligenceoperations with respect to the objects, and so forth. Because theseoperations are performed at the provider level, the media player devicesoperated by users may not need to be associated with particularlypowerful computing resources, which conserves user device resources,provides convenience to users (e.g., in terms of portability, cooling,etc.), and enables various types of media player devices (e.g., withvarious form factors, various price points, etc.) to provide theexperience of the virtual scene to users.

While certain examples described herein may reference a few specificarticles from pluralities that may have any suitable numbers of thearticles, it will be understood that the same principles described inaccordance with the few specific articles may apply to one or more otherarticles in the pluralities, up to and including all of the articles ineach respective plurality of articles. For instance, due to convenienceand clarity of description, a few articles may be designated by ordinaladjectives such as “first,” “second,” and the like (e.g., first andsecond virtual vantage points, first and second 3D rendering engines,first and second surface data frames, etc.). However, as will generallybe illustrated in the figures, principles described herein may apply tomany or all of the articles included in a plurality of the articles, asopposed to just the first and second articles, for example. Thus, aswill be described below, certain implementations may include many (i.e.,more than two) 3D rendering engines rendering many surface data frameseach associated with the view from one of many virtual vantage points,and so forth.

Various embodiments will now be described in more detail with referenceto the figures. The disclosed methods and systems may provide one ormore of the benefits mentioned above and/or various additional and/oralternative benefits that will be made apparent herein.

FIG. 1 illustrates an exemplary virtual scene capture system 100(“system 100”) for rendering frames of a virtual scene from differentvantage points based on a virtual entity description frame of thevirtual scene. As shown, system 100 may include, without limitation, avirtual entity state tracking facility 102, a virtual entity descriptionframe facility 104, and a storage facility 106 selectively andcommunicatively coupled to one another. It will be recognized thatalthough facilities 102 through 106 are shown to be separate facilitiesin FIG. 1, facilities 102 through 106 may be combined into fewerfacilities, such as into a single facility, or divided into morefacilities as may serve a particular implementation. In some examples,each of facilities 102 through 106 may be distributed between multipledevices and/or multiple locations as may serve a particularimplementation. Each of facilities 102 through 106 will now be describedin more detail.

Virtual entity state tracking facility 102 may include one or morephysical computing devices (e.g., hardware and/or software componentssuch as processors, memories, communication interfaces, instructionsstored in memory for execution by the processors, etc.) that performvarious operations associated with rendering frames of a virtual scenefrom different vantage points based on a virtual entity descriptionframe of the virtual scene. For example, using the one or more physicalcomputing devices, virtual entity state tracking facility 102 maymaintain data representative of a plurality of virtual entities includedwithin a virtual 3D space of a virtual scene. Virtual entity statetracking facility 102 may maintain the data in any suitable way. Forexample, virtual entity state tracking facility 102 may receive, track,generate, analyze, organize, and/or otherwise process datarepresentative of the plurality of virtual entities of the virtualscene. As will be described in more detail below, virtual entity statetracking facility 102 may also receive commands to modify the maintaineddata (e.g., to modify one or more of the virtual entities such as byadding, removing, replacing, moving, rotating, enlarging, or otherwisemodifying the virtual entities) and may implement the commands bymodifying the data being maintained. Virtual entity state trackingfacility 102 may further maintain the data by interoperating withstorage facility 106 to store data representative of each virtual entityin storage facility 106.

As used herein, a “virtual entity” may refer to any virtual item thatmay be associated with a virtual scene and/or a virtual 3D space. Forexample, among the virtual entities for which virtual entity statetracking facility 102 maintains data, the virtual 3D space of thevirtual scene may include virtual entities such as one or more virtualobjects, a plurality of virtual vantage points into the virtual 3D space(e.g., virtual capture devices positioned and angled in particular wayswith respect to the virtual 3D space so as to capture the virtual 3Dspace from a variety of different perspectives), and/or any othervirtual entities as may serve a particular implementation. Inparticular, as will be described and illustrated below, one exemplaryvirtual 3D space may include a virtual object surrounded by a pluralityof virtual vantage points including a first virtual vantage point and asecond virtual vantage point different from the first virtual vantagepoint.

Virtual entity description frame facility 104 may include one or morephysical computing components (e.g., hardware and/or software componentsseparate from those of virtual entity state tracking facility 102 orshared with virtual entity state tracking facility 102) that performvarious operations associated with generating and/or providing virtualentity description frames to be used for rendering frames of the virtualscene from the plurality of vantage points including the first andsecond virtual vantage points. For example, using the one or morephysical computing devices, virtual entity description frame facility104 may generate (e.g., based on the data representative of theplurality of virtual entities maintained by virtual entity statetracking facility 102) a virtual entity description frame representativeof a state of at least one virtual entity (e.g., and, in some examples,all of the virtual entities) in the plurality of virtual entities at aparticular point in a temporal sequence (e.g., a particular moment inreal time, a particular point representing a moment on a virtualtimeline unrelated to real time, etc.).

As used herein, a “virtual entity description frame” may refer to adataset (e.g., including object description data represented in alanguage such as Java Script Object Notation (“JSON”) or the like) thatdescribes a state of one or more virtual entities included in a virtual3D space of a virtual scene. For example, a virtual entity descriptionframe may include data describing each of several virtual entitiesincluded in the virtual 3D space at a particular point in a temporalsequence. For instance, the virtual entity description frame may includestate data representative of a location where each virtual entity ispositioned with respect to a global coordinate system associated withthe virtual 3D space, angles and orientations at which each virtualentity is positioned, relative sizes of each virtual entity, one or moremovement vectors for each virtual entity, colors and/or textures forvarious surfaces of each virtual entity, and/or any other state datathat may be used to describe particular virtual entities at theparticular point in the temporal sequence as may serve a particularimplementation. Exemplary virtual entity description frames will bedescribed in more detail below.

Once virtual entity description frame facility 104 has generated thevirtual entity description frame, virtual entity description framefacility 104 may provide the virtual entity description frame to aplurality of server-side 3D rendering engines associated with a contentprovider system. As used herein, “server-side” may refer to a serverside (e.g., a provider's side) of a server-client transaction such as atransaction where a content provider system provides content (e.g.,virtual reality media content) to a client device used by an end user.For example, as will be described in more detail below, a virtualreality media content provider system may provide virtual reality mediacontent to a media player device associated with a user. As such,server-side systems and components may refer to those systems andcomponents that are associated with (e.g., included within, implementedby, interoperate with, etc.) the content provider system to provide data(e.g., virtual reality media content) to the media player device (e.g.,by way of a network). In contrast, “client-side” devices may beassociated with the client device (e.g., the media player device) usedby the user on the other side of the network, and may include devicesthat facilitate the client device with receiving the data from thecontent provider system (e.g., the media player device and/or othercomputer components operated by the user on the user's side of thenetwork).

Accordingly, 3D rendering engines may be implemented on the server sideof the network (i.e., associated with system 100 and/or other elementsof a content provider system) by hardware and/or software resources thatmay be integrated with or separate from and communicatively coupled tothe hardware and/or software resources of system 100. The 3D renderingengines may be configured to render, based on a virtual entitydescription frame, respective surface data frames associated withparticular virtual vantage points. For example, virtual entitydescription frame facility 104 may provide the virtual entitydescription frame to a first 3D rendering engine associated with thefirst virtual vantage point and configured to render (e.g., based on thevirtual entity description frame) a first surface data framerepresentative of color and depth data of surfaces of a virtual objectvisible from the first virtual vantage point at the particular point inthe temporal sequence. Additionally, virtual entity description framefacility 104 may provide the same virtual entity description frame to asecond 3D rendering engine associated with the second virtual vantagepoint and configured to render (e.g., also based on the virtual entitydescription frame) a second surface data frame representative of colorand depth data of surfaces of the virtual object visible from the secondvirtual vantage point at the particular point in the temporal sequence.

As used herein, a “surface data frame” may refer to a dataset thatrepresents various types of data associated with surfaces of objects(e.g., virtual objects) visible within a virtual scene from a particularvantage point and at a particular point in a temporal sequenceassociated with the virtual scene. For example, a surface data frame mayinclude color data (i.e., image data) as well as depth datarepresentative of the objects as viewed from a particular vantage pointwith respect to the virtual scene. As such, a plurality of relatedsurface data frames may be sequenced together to create a video-likerepresentation (representing not only color but also depth data) of thevirtual scene as the virtual scene would be viewed or experienced fromthe particular vantage point. In certain examples, a surface data framemay further be associated with other types of data such as audio data,metadata (e.g., metadata including information about specific objectsrepresented in the surface data frame and/or information about vantagepoints associated with the virtual scene), and/or other types of data asmay serve a particular implementation. Examples of surface data framesassociated with different vantage points, as well as sequences ofrelated surface data frames will be described and illustrated below.

As used herein, “color data” may broadly include any image data, videodata, or the like, whether represented in color or grayscale (i.e.,“black and white”), that represents how a subject (e.g., a virtualobject included within a virtual 3D space of a virtual scene) may appearat a particular point in a temporal sequence or over a particular timeperiod from the perspective of a particular vantage point. Color data isnot limited to any particular format, file type, frame rate, resolution,quality level, or other characteristic that may be associated withvarious definitions and/or standards defining image data and/or videodata in the art. Similarly, as used herein, “depth data” may include anydata representative of a position of a subject in space. For example,depth data representative of a virtual object may include coordinateswith respect to a global coordinate system (e.g., a global coordinatesystem associated with the virtual 3D space of the virtual scene) fordifferent points on the surfaces of the virtual object.

Storage facility 106 may maintain any suitable data received, generated,managed, tracked, maintained, used, and/or transmitted by facilities 102or 104 in a particular implementation. For example, as shown, storagefacility 106 may include virtual object data 108, which may include data(e.g., state data) associated with one or more virtual objects includedwithin a virtual 3D space of a virtual scene, as well as virtual vantagepoint data 110, which may include data (e.g., state data) associatedwith one or more virtual vantage points into the virtual 3D space.Additionally, storage facility 106 may include data associated withother types of virtual entities included within the virtual 3D space ofthe virtual scene, instructions (e.g., programming instructions) forperforming the operations described herein, and/or any other data as mayfacilitate facilities 102 and 104 in performing the operations describedherein. For example, storage facility 106 may further include data(e.g., object description data, color data, depth data, audio data,metadata, etc.) associated with surface data frames, virtual entitydescription frames, and the like. Storage facility 106 may also maintainadditional or alternative data as may serve a particular implementation.

In certain examples, system 100 may be associated with various otherserver-side systems (e.g., virtual scene control systems, asset storagesystems, video data packaging systems, 3D rendering engines, etc.)included together in various configurations within a content providersystem (e.g., a virtual reality media content provider system) in orderto render surface data frames of a virtual scene from different vantagepoints and to provide the surface data frames (e.g., as part of virtualreality media content) to be presented to a user to allow the user toexperience the virtual scene.

In some implementations, it will be understood that one or more of theseother server-side systems may be integrated with (e.g., included within)system 100 or otherwise closely associated with system 100 (e.g.,communicatively coupled to system 100, operated by the same or relatedvirtual reality media provider entities, etc.). For example, in aparticular implementation, system 100 may include an asset storagesystem storing color and depth data representative of one or morevirtual objects, a plurality of 3D rendering engines communicativelycoupled to the asset storage system, and a virtual entity state trackingsystem communicatively coupled to the asset storage system and/or to the3D rendering engines. The entity state tracking system may be configuredto perform one or more of the operations described above in relation tofacilities 102 through 106. In other implementations, system 100 may beimplemented as a separate, standalone system that is not integrated withthese other server-side systems but, rather, is communicatively coupledto the other server-side systems and/or otherwise configured tointeroperate with the other server-side systems as may serve aparticular implementation.

By way of illustration, FIG. 2 shows an exemplary configuration 200 inwhich system 100 facilitates rendering frames of a virtual scene fromdifferent vantage points based on a virtual entity description frame ofthe virtual scene. As shown in FIG. 2, an implementation of system 100may be communicatively coupled to a plurality of virtual scene controlsystems 202 (e.g., virtual scene control systems 202-1 through 202-M) aswell as to a plurality of server-side 3D rendering engines 204 (e.g., 3Drendering engines 204-1 through 204-N). For example, system 100 may becommunicatively coupled to virtual scene control systems 202 and/or to3D rendering engines 204 by way of one or more networks (e.g., includingany of the networks or network technologies described herein) or by wayof other modes of communication as may serve a particularimplementation. As shown in configuration 200, a virtual entity statetracking system that performs the operations described above in relationto facilities 102 through 106 may be implemented by system 100. Asmentioned above, in other implementations, system 100 may embody both anentity tracking system configured to perform these operations and one ormore of the other systems and devices illustrated in configuration 200.

System 100 may provide, via the communicative connection with 3Drendering engines 204, one or more virtual entity description framesincluding a virtual entity description frame 206. Based on virtualentity description frame 206 as well as data requested and received froman asset storage system 208 that is communicatively coupled with 3Drendering engines 204, 3D rendering engines 204 may each renderrespective surface data frames 210 (e.g., surface data frames 210-1through 210-N) and may provide surface data frames 210 to a video datapackaging system 212. System 100 has been described in detail above withrespect to FIG. 1. Each of the other systems and items illustrated inconfiguration 200 will now be described in more detail.

Virtual scene control systems 202 may represent any computing systemsconfigured to request and/or otherwise implement changes to one or morevirtual entities included in a virtual 3D space of a virtual scene(e.g., virtual entities about which data is maintained by system 100).For example, one or more virtual scene control systems 202 (e.g.,virtual scene control system 202-1) may be associated with (e.g.,maintained by, operated by, etc.) a content creator responsible fororiginally generating the data representative of the virtual entitiesincluded within the virtual 3D space of the virtual scene. Additionally,in certain implementations, one or more other virtual scene controlsystems 202 (e.g., virtual scene control system 202-2) may be associatedwith an end user that is experiencing the virtual 3D space of thevirtual scene. For example, virtual scene control system 202-2 may beimplemented by a media player device currently rendering the virtualentities to allow a user of the media player device to experience andinteract with the virtual entities within the virtual 3D space of thevirtual scene.

Because system 100 may maintain one unified set of data representativeof all the virtual entities included within the virtual 3D space (e.g.,as opposed to separate sets of data representative of the virtualentities for each virtual scene control system 202), as each of virtualscene control systems 202 makes modifications to the virtual entities,those modifications may be reflected in the unified set of data.Accordingly, multiple users (i.e., different users associated withdifferent virtual scene control systems 202) may all effectmodifications to the same virtual 3D space of the same virtual scene. Asa result, the modifications made by all of virtual scene control systems202 may be reflected in virtual entity description frames output bysystem 100 (e.g., virtual entity description frame 206), and may, inturn, be reflected in each of the surface data frames rendered by 3Drendering engines 204 (e.g., surface data frames 210).

To illustrate how virtual scene control systems 202 may modify thevirtual entities in a virtual 3D space, FIG. 3 shows an exemplaryvirtual scene 300 including a plurality of virtual entities, and FIG. 4illustrates exemplary modifications that may be made to virtual scene300. Specifically, referring first to FIG. 3, virtual scene 300 isassociated with a virtual 3D space 302 that includes a virtual object304 and is surrounded by a plurality of virtual vantage points 306(e.g., virtual vantage points 306-1 through 306-8).

Virtual scene 300 may represent any type of scene (e.g., a real-worldscene, a computer-generated scene, an event, etc.) as may serve aparticular implementation. As illustrated by the circle, the virtual 3Dspace 302 associated with virtual scene 300 may be a specificallydelineated area such as a stage, an arena, or the like. Conversely, inother examples, virtual 3D space 302 may not be so well defined ordelineated. For example, virtual 3D space 302 may represent any indooror outdoor location (e.g., based on the real world or based on animaginary or computer-generated world), event, landscape, structure, orthe like.

Virtual object 304 may represent any virtual object, whether living orinanimate, that is associated with (e.g., located within or around)virtual 3D space 302 and that is detectable (e.g., viewable, etc.) fromat least one of virtual vantage points 306. For example, virtual object304 may be based on a real-world object (e.g., an object for which a 3Dmodel has been generated), an imaginary or computer-generated object, orthe like. While virtual object 304 is drawn as a relatively simplegeometric shape for the sake of clarity, it will be understood thatvirtual object 304 may represent various types of objects having variouslevels of complexity. Rather than a geometric shape, for instance,virtual object 304 could represent any animate or inanimate object orsurface, such as a person or another living thing, a non-transparentsolid, liquid, or gas, a less discrete object such as a wall, a ceiling,or a floor, or any other type of object described herein or as may servea particular implementation. As shown, virtual object 304 may includevarious surfaces such that virtual object 304 may look different whenviewed from each different virtual vantage point 306, as will beillustrated below.

Along with virtual object 304, virtual scene 300 also includes virtualvantage points 306 into virtual 3D space 302. As used herein, a virtualvantage point “into” a virtual 3D space may refer to a virtual vantagepoint that is positioned, angled, oriented, etc., with respect to thevirtual 3D space in any suitable way. For example, a virtual vantagepoint into a virtual 3D space may be a virtual vantage point that isincluded within the virtual 3D space, is outside of the virtual 3D spacewith a perspective looking into the virtual 3D space, is surrounding thevirtual 3D space along with other virtual vantage points, and/or isotherwise associated with the virtual 3D space in any suitable way so asto provide a view of at least some portion of the virtual 3D space.

As shown, each virtual vantage point 306 may be represented in FIG. 3with a labeled circle disposed at a particular location with respect tovirtual 3D space 302 and that has dotted lines emanating therefrom toillustrate a field of view associated with the virtual vantage point306. The positions associated with virtual vantage points 306 may befixed with respect to virtual 3D space 302, although, as will bedescribed below, it may be possible for the fixed positions to bemodified by one of virtual scene control systems 202. Additionally, insome examples, it will be understood that both virtual 3D space 302 andvirtual vantage points 306 may be moving through virtual scene 300together (e.g., such as a vehicular virtual 3D space like a spaceship, ahot air balloon, or the like). As shown, the fixed positions at whichvirtual vantage points 306 are disposed may, in some examples, surroundvirtual 3D space 302 along at least two dimensions associated withvirtual 3D space 302 (e.g., along a plane such as the ground). In otherexamples, positions 306 may further surround virtual 3D space 302 alongthree dimensions (e.g., by including positions 306 above and below 302as well). Even in examples where virtual vantage points 306 surroundvirtual 3D space 302 along only two dimensions, pluralities of virtualvantage points 306 may be “stacked” at different heights relative to thepositions encircling virtual 3D space 302 shown in FIG. 3 in order toview virtual object 304 from related but slightly differentperspectives.

While each of virtual vantage points 306 illustrated in FIG. 3 areangled inwardly toward virtual 3D space 302 so as to capture virtual 3Dspace 302 from various angles to enable virtual 3D space 302 to later berendered from arbitrary virtual vantage points, it will be understoodthat, in certain examples, one or more of virtual vantage points 306 maybe angled outwardly (i.e., away from virtual 3D space 302) to viewvirtual objects surrounding virtual 3D space 302 or the like. Forinstance, a 360-degree virtual vantage point may be positioned in themiddle of virtual 3D space 302 (not explicitly shown) to provide datarepresentative of virtual objects included within virtual 3D space 302from additional perspectives and/or data representative of virtualobjects outside of virtual 3D space 302.

As mentioned above, FIG. 4 illustrates exemplary modifications that maybe made to virtual scene 300. Specifically, in some examples, system 100may receive a command to modify the maintained data representative ofthe plurality of entities (i.e., data representative of virtual object304, virtual vantage points 306, and/or any other virtual entitiesincluded in virtual 3D space 302), and, in response to the receiving ofthe command, may modify the maintained data representative of theplurality of virtual entities in accordance with the command. Forexample, the command may be sent (e.g., by way of a web socket oranother suitable type of communication) by any of virtual scene controlsystems 202 using JSON code or another suitable object description codedescribing the modification that is to be made.

The virtual entities included within virtual scene 300 may be modifiedin any suitable manner, which may be determined in part by the type ofthe virtual entity being modified. For example, if the virtual entitybeing modified is a virtual object, the modifying of the maintained datarepresentative of the plurality of virtual entities in accordance withthe command may include adding an additional virtual object to theplurality of virtual entities. Additionally or alternatively, themodifying may include replacing the virtual object included within theplurality of virtual entities with an additional virtual object,removing the virtual object from the plurality of virtual entities,modifying at least one property of a virtual object included in theplurality of virtual entities, and/or otherwise modifying the virtualobject with respect to other virtual entities and/or with respect to thevirtual 3D space of the virtual scene.

To illustrate, FIG. 4 shows an additional virtual object 402 that isadded to virtual 3D space 302 along with virtual object 304. It will beunderstood that in other examples, virtual object 402 could insteadreplace virtual object 304 (i.e., such that virtual object 304 isremoved from virtual 3D space 302 while virtual object 402 is added tovirtual 3D space 302). As further shown in FIG. 4, certain properties ofvirtual object 304 (e.g., the position and orientation of virtual object304) may be modified. In other examples, other properties such as thesize, color, texture, posture, and/or any other properties of virtualobject 304 could similarly be modified.

If the virtual entity being modified is a virtual vantage point (e.g.,one of virtual vantage points 306), the modifying of the maintained datarepresentative of the plurality of virtual entities in accordance withthe command may include adding an additional virtual vantage point tothe plurality of virtual entities. Additionally or alternatively, themodifying may include modifying at least one of the plurality of virtualvantage points included within the plurality of virtual entities,removing at least one of the plurality of virtual vantage points fromthe plurality of virtual entities, or the like. For example, a field ofview associated with one of virtual vantage points 306 (e.g., virtualvantage point 306-1) may be changed or turned to get a perspective on adifferent angle of virtual 3D space 302. In other examples, virtualvantage points 306 may be moved inward or outward (e.g., to create azoomed in or zoomed out effect with respect to a particular virtualobject within virtual 3D space 302), removed from the plurality ofvirtual vantage points 306, or otherwise modified. As another example,an additional virtual vantage point may be added to the plurality ofvirtual vantage points 306 to get another perspective on virtual objects304 and 402 (e.g., a perspective that is not well covered by one ofvirtual vantage points 306-1 through 306-8).

As described above, in some examples, a virtual object such as virtualobject 304 may be modified (e.g., moved and/or rotated with respect tovirtual 3D space 302) based on a direct command from one of virtualscene control systems 202 to modify the virtual object. In otherexamples, however, a virtual object may be modified automatically (i.e.,modified in the same or different ways but without being based on anexplicit command from a virtual scene control system 202) based oninteractions with other virtual entities included within virtual 3Dspace 302. More specifically, for example, the maintaining by system 100of the data representative of the plurality of virtual entities mayinclude applying (e.g., to the virtual objects included within thevirtual 3D space of the virtual scene) at least one of a physics-basedobject behavior and an artificial intelligence-based (Al-based) objectbehavior.

For instance, a physics-based object behavior 404 is illustrated in FIG.4. When a modification to add virtual object 402 to virtual 3D space 302is made (e.g., by system 100), system 100 may determine that virtualobjects 304 and 402 each represent solid virtual objects that cannotexist in the same virtual space. Accordingly, as illustrated byphysics-based object behavior 404, locational and orientationalproperties of virtual object 304 may be modified in accordance withphysics rules such that virtual object 402 partially displaces virtualobject 304 (i.e., “bumps” virtual object 304 out of the way). Otherphysics-based object behaviors may mimic other rules of physics (e.g.,real-world physics or imaginary physics that apply only in the virtualworld) that define how objects interact with one another and withphysical forces and principles (e.g., gravity, momentum, friction,buoyancy, light reflection, etc.). These physics-based object behaviorsmay also be applied to the maintained data representative of virtualobjects included within virtual 3D space 302 by system 100. Moreover,Al-based object behaviors may also help define how virtual objectsinteract with one another and with the environment in which the virtualobjects are placed. For example, Al-based object behaviors may beparticularly applicable with virtual objects (e.g., avatars)representing living things such as people and/or animals who may useartificial intelligence to make “choices” such as where to walk withinvirtual 3D space 302, who to talk to and what to say, when to run fromdanger, and so forth.

Returning to FIG. 2, system 100 generates virtual entity descriptionframes representative of the states of the virtual entities in theplurality of virtual entities at particular points in a temporalsequence (e.g., a real time sequence, a virtual timeline associated withtime in a virtual world, etc.). For example, as shown, system 100 maygenerate a particular virtual entity description frame (i.e., virtualentity description frame 206), and may provide virtual entitydescription frame 206 to each of 3D rendering engines 204. 3D renderingengines 204 may be server-side 3D rendering engines (e.g., 3D renderingengines across a network and/or otherwise separated from client-sidedevices such as media player devices used by users). In some examples,3D rendering engines 204 may be implemented by separate devices (e.g.,separate servers, separate processors within a server, etc.) or byseparate software processes (e.g., separate instruction threads, etc.),while in other examples, 3D rendering engines 204 may be integratedtogether into common hardware and/or software devices or processes asmay serve a particular implementation. In some implementations, 3Drendering engines may be jointly operated with or even fully integratedinto a virtual scene capture system such as system 100, while in otherimplementations 3D rendering engines may be operated separately (e.g.,by a different entity providing cloud-based processing services or thelike).

Certain virtual entity description frames provided to 3D renderingengines 204 may be key description frames that include state datarepresentative of all the virtual entities associated with the virtualscene (i.e., virtual scene 300) at a particular point in the temporalsequence, while other virtual entity description frames may be updatedescription frames representative of a state (e.g., at a particularpoint in the temporal sequence) of change of only those virtual entitiesassociated with the virtual scene that have been modified since aprevious key description frame was generated representing the state ofall the virtual entities at a previous point in the temporal sequence.For example, referring to the modifications illustrated in FIG. 4, a keydescription frame may include state data associated with virtual objects304 and 402, as well as state data associated with virtual vantagepoints 306. In contrast, an update description frame may include onlystate data associated with virtual objects 304 and 402 or only statedata associated with changes to virtual objects 304 and 402, because,for example, virtual objects 304 and 402 may have been modified since aprevious key description frame was generated. Data representative of thestate of virtual vantage points 306 may not be represented in thisexemplary update description frame because virtual vantage points 306may have remained statically positioned and unchanged since the previouskey description frame.

FIG. 5 shows a plurality of exemplary virtual entity description frames500 (e.g., virtual entity description frames 500-1 through 500-12) thatmay be generated by system 100. As indicated by arrows pointing from onevirtual entity description frame 500 to another, virtual entitydescription frames 500 may be ordered in a temporal sequence startingwith virtual entity description frame 500-1 and progressing to virtualentity description frame 500-12, after which the temporal sequence mayprogress to additional virtual entity description frames 500 that arenot explicitly shown in FIG. 5. Along the bottom of each virtual entitydescription frame 500, the type of virtual entity description frame(e.g., key description frame or update description frame) is indicated.Specifically, virtual entity description frames 500-1, 500-5, and 500-9are indicated as being key description frames, while virtual entitydescription frames 500-2 through 500-4, 500-6 through 500-8, and 500-10through 500-12 are indicated as being update description frames.

Accordingly, in this example, each key description frame is followed inthe temporal sequence by several (e.g., three) update descriptionframes, which are in turn followed in the temporal sequence by anotherkey description frame. It will be understood, however, that thearrangement of key description frames and update description framesshown in FIG. 5 is exemplary only, and that the arrangement of key andupdate description frames may be implemented in any way as may serve aparticular implementation. For example, a virtual scene that is notparticularly dynamic (i.e., is not affected by a large number ofmodifications to virtual entities) may be represented by relatively fewkey description frames followed by relatively large numbers of updatedescription frames. Conversely, a virtual scene that is more dynamic maybe represented by a larger proportion of key description frames (up toand including exclusively key description frames) and a smallerproportion of update description frames (down to and including no updatedescription frames).

As further shown in FIG. 5, each virtual entity description frame 500may include or be implemented by virtual entity description code (e.g.,JSON code, XML code, or another type of code suitable for describingstate data associated with the virtual entities maintained by system100) and may be associated with a sequence number (e.g., anidentification number or “ID”) indicative of a position of therespective virtual entity description frame 500 in the temporal sequencewith respect to other virtual entity description frames 500. Forexample, as shown, virtual entity description frame 500-1 may have asequence number that is a whole number (i.e., “1.0”) to indicate thatvirtual entity description frame 500-1 is a key description frame and toindicate the relative position of the frame with respect to other keydescription frames (e.g., “1.0” comes before “2.0”). Virtual entitydescription frames 500-2 through 500-4 may then each be associated withsequence numbers that begin with a 1 (i.e., to indicate that theseframes are updates to key description frame 1.0) and includesub-identifiers (i.e., “0.1,” “0.2,” and “0.3”) to indicate the relativepositions of the update description frames in the temporal sequence withrespect to other update description frames (e.g., “1.1” comes before“1.2”). This virtual entity description frame numbering scheme isexemplary only and any suitable frame numbering scheme may be employedas may serve a particular implementation.

Returning to FIG. 2, regardless of whether virtual entity descriptionframe 206 is a key description frame (e.g., such as virtual entitydescription frames 500-1, 500-5, or 500-9) or an update descriptionframe (e.g., such as the other virtual entity description frames 500 inFIG. 5), the sequence of virtual entity description frames includingvirtual entity description frame 206 may provide all the informationneeded by 3D rendering engines 204 to render virtual 3D space 302 ofvirtual scene 300 from the respective vantage points with which each 3Drendering engine 204 is associated. As such, it may not be necessarythat 3D rendering engines 204 receive or process virtual entitydescription frames in order. Rather, 3D rendering engines 204 may renderrespective surface data frames 210 (e.g., which may each be generatedfrom a single virtual entity description frame in the sequence) in anyorder as may be convenient or efficient for a particular 3D renderingengine 204, and the surface data frames 210 may be reordered andsynchronized later (e.g., by video data packaging system 212).

In some examples, virtual entity description frame 206 may include stateinformation representative of the virtual entities along with links todetailed information (e.g., binary data representative of virtual objectgeometries, textures, etc.) that is stored in asset storage system 208and may be accessed, based on the links in virtual entity descriptionframe 206, from asset storage system 208 by each of 3D rendering engines204 as needed. Asset storage system 208 may be implemented by a separatedevice from system 100 and/or 3D rendering engines 204 (e.g., a separateserver, a separate processor and storage facility within a server,etc.), by separate software processes (e.g., separate instructionthreads, etc.), or may be integrated together into common hardwareand/or software devices or processes with system 100 and/or 3D renderingengines 204 as may serve a particular implementation. In someimplementations, asset storage system 208 may be jointly operated withor fully integrated into a virtual scene capture system such as system100 and/or into a system that also includes 3D rendering engines 204,while in other implementations asset storage system 208 may be operatedseparately (e.g., by a different entity providing cloud-based processingservices or the like).

In any case, between data included within virtual entity descriptionframe 206 and data accessed from asset storage system 208 using linksprovided within virtual entity description frame 206, 3D renderingengines 204 may be receive access to all the information necessary torender surface data frames 210 representing virtual 3D space 302 fromrespective virtual vantage points without having to rely on informationmaintained locally by 3D rendering engines 204.

Each 3D rendering engine 204 may be associated with one of the virtualvantage points represented in the plurality of virtual entitiesmaintained by system 100. For example, 3D rendering engines 204-1through 204-8 (of which only 3D rendering engines 204-1 and 204-2 areexplicitly shown in FIG. 2) may be associated with virtual vantagepoints 306-1 through 306-8 (illustrated in FIG. 3), respectively. Assuch, each 3D rendering engine 204 may render a respective surface dataframe 210 as seen from the perspective (i.e., the position, angle, fieldof view, etc.) of the virtual vantage point 306 with which theparticular 3D rendering engine 204 is associated. Moreover, as describedabove, each surface data frame 210 may include not only color data(i.e., image data) representative of the appearance of virtual objectsfrom a respective virtual vantage point, but may also include depthdata.

To illustrate, FIG. 2 shows images representative of surface data frame210-1, which may be a surface data frame rendered by 3D rendering engine204-1, the 3D rendering engine associated with virtual vantage point306-1 (see FIG. 3). As shown, surface data frame 210-1 may include colordata 214, which may represent a view of virtual 3D space 302 (includingcolor data from the surfaces of virtual object 304) visible from virtualvantage point 306-1. While color data 214 is illustrated as an image inFIG. 2, it will be understood that color data 214 may be captured,encoded, formatted, transmitted, and represented in any suitable form.For example, color data may be digital data that is formatted accordingto a standard video encoding protocol, a standard image format, or thelike. Color data 214 may represent a color image (e.g., similar to acolor photograph) of the virtual objects included within virtual 3Dspace 302 as viewed from virtual vantage point 306-1. Additionally oralternatively, color data 214 may be a grayscale image representative ofthe virtual objects (e.g., similar to a black and white photograph).

Additionally, surface data frame 210-1 may include depth data 216, whichmay represent another view of virtual 3D space 302 that includes depthdata for the surfaces of virtual object 304 from a point in spaceassociated with virtual vantage point 306-1. Like color data 214, depthdata 216 may depict virtual object 304 within virtual 3D space 302 fromthe perspective of virtual vantage point 306-1. However, rather thanrepresenting the visible appearance of virtual object 304 (i.e.,representing in color or grayscale how light interacts with the surfacesof virtual object 304), depth data 216 may represent the depth (i.e.,the distance or position) of each point on the surface of virtual object304 (e.g., as well as other objects within virtual 3D space 302)relative to the virtual position of virtual vantage point 306-1. As withcolor data 214, depth data 216 may be captured, encoded, formatted,transmitted, and represented in any suitable form. For example, asshown, depth data 216 may be represented by grayscale image data (e.g.,six or eight bits for each pixel represented within depth data 216).However, rather than representing how visible light reflects from thesurfaces of virtual object 304 (i.e., as represented in color data 214),the grayscale image of depth data 216 may represent, for each pixel inthe image, how far away the point represented by that pixel is fromvirtual vantage point 306-1. For example, points that are closer tovirtual vantage point 306-1 may be represented with values thatrepresent darker shades of gray (e.g., binary values closer to 0b111111in the case of a six-bit implementation where 0b111111 representsblack). Conversely, points that are farther away from virtual vantagepoint 306-1 may be represented with values that represent lighter shadesof gray (e.g., binary values closer to 0b000000 in the case of thesix-bit implementation where 0b000000 represents white).

Respective sets of surface data frames 210 may be generated by 3Drendering engines 204 such that each virtual entity description frameprovided by system 100 (e.g., including virtual entity description frame206) is associated with a respective set of surface data frames (e.g., aset that includes surface data frames 210-1 through 210-N) representingrenderings of virtual 3D space 302 of virtual scene 300 from differentvirtual vantage points into virtual 3D space 302. As shown in FIG. 2,each surface data frame 210 in the respective sets of surface dataframes may then be provided to video data packaging system 212, whichmay organize, synchronize, encode, compress, combine, and/or otherwiseprocess the surface data frames to generate respective color video datastreams and depth video data streams associated with each virtualvantage point 306.

To illustrate, FIG. 6 shows a more detailed view of certain componentsof configuration 200. Specifically, FIG. 6 illustrates eight 3Drendering engines 204 (i.e., 3D rendering engines 204-1 through 204-8)that render a complete set of surface data frames 210 (i.e., surfacedata frames 210-1 through 210-8). Surface data frames 210-1 through210-8 may be representative of color and depth data of surfaces ofvirtual objects included within a virtual 3D space (e.g., virtual object304 within virtual 3D space 302) as the surfaces would appear fromdifferent vantage points with respect to the virtual 3D space (e.g.,virtual vantage points 306-1 through 306-8, respectively). It will beunderstood that, while surface data frames may each be represented inFIG. 6 by an image analogous to the color data image illustrated in FIG.2 (i.e., the image illustrating color data 214), each surface data frame210 may also include data representative of depth data, which may berepresented by an image analogous to the depth data image illustrated inFIG. 2 (i.e., the image illustrating depth data 216).

3D rendering engines 204 may respectively generate surface data frames210 from the associated virtual vantage points 306 based on virtualentity description frame 206, as well as based on data accessed fromasset storage system 208, as described above. For example, themaintained data representative of the plurality of virtual entitiesassociated with virtual scene 300 may include a link to color and depthdata representative of virtual object 304 that is stored in assetstorage system 208. As such, virtual entity description frame 206 (whichmay have been generated by system 100 and provided to 3D renderingengines 204 as described above) may be generated to include the link tothe color and depth data representative of virtual object 304 stored inasset storage system 208. Each of 3D rendering engines 204 may beconfigured to render their respective surface data frames 210 from theirrespective virtual vantage points 306 by performing operations includingreceiving virtual entity description frame 206 from system 100, andaccessing the color and depth data representative of virtual object 304stored in asset storage system 208 from asset storage system 208 usingthe link included within virtual entity description frame 206. With boththe data included within virtual entity description frame 206 and thecolor and depth data accessed from asset storage system 208, 3Drendering engines 204 may render surface data frames 210 to provideviews of virtual 3D space 302 (e.g., including virtual object 304) fromvantage points surrounding virtual 3D space 302, as shown.

Each surface data frame 210 may be included within a separate sequenceof surface data frames representative of color and depth data of thesurfaces of virtual object 304 visible from the respective virtualvantage point 306 during the temporal sequence. For example, surfacedata frame 210-1 may be included within a first sequence of surface dataframes representative of color and depth data of the surfaces of virtualobject 304 visible from virtual vantage point 306-1 during the temporalsequence, surface data frame 210-2 may be included within a secondsequence of surface data frames representative of color and depth dataof the surfaces of virtual object 304 visible from virtual vantage point306-2 during the temporal sequence, and so forth. In other words, oncesurface data frames 210 have been rendered, each 3D rendering engine 204may continue to render other surface data frames in different respectivesequences of surface data frames. For example, 3D rendering engines 204may receive additional virtual entity description frames after virtualentity description frame 206 (e.g., such as the sequence of virtualentity description frames 500 illustrated in FIG. 5), and may generatefurther surface data frames based upon the additional virtual entitydescription frames.

To illustrate, FIG. 7 shows a plurality of exemplary frame sequences 702(e.g., frame sequences 702-1 through 702-8) of surface data framesrepresentative of color and depth data of surfaces of virtual object 304visible from vantage points 306-1 through 306-8, respectively. Forexample, the first surface data frames in each frame sequence 702 (i.e.,the surface data frames that are uncovered so that different views ofobject 304 are visible in FIG. 7) may correspond to surface data frames210 illustrated in FIG. 6. Accordingly, frame sequences 702-1 through702-8 may be associated, respectively, with 3D rendering engines 204-1through 204-8 and, thus, with virtual vantage points 306-1 through306-8. For example, frame sequence 702-1 may represent both the colorand the depth of virtual objects included within virtual 3D space 302 asviewed from virtual vantage point 306-1 during a particular temporalsequence 704 (e.g., a particular period of real time, a particularvirtual timeline associated with an immersive virtual reality world,etc.). Similarly, frame sequence 702-2 may represent the color and depthof virtual objects included within virtual 3D space 302 as viewed fromvirtual vantage point 306-2 during temporal sequence 704, and so forthfor frame sequences 702-3 through 702-8.

As described and illustrated above, each of the surface data framesgenerated by 3D rendering engines 204 and included in frame sequences702 may be transmitted or otherwise passed into video data packagingsystem 212, which may be communicatively coupled to 3D rendering engines204. Based on each of the different frame sequences 702 of surface dataframes, video data packaging system 212 may generate one or moretransport streams to transmit a color video data stream and a depthvideo data stream for each of virtual vantage points 306. For example,video data packaging system 212 may generate a single transport streamthat contains individual color video data streams and depth video datastreams associated with each frame sequence 702 (i.e., associated witheach 3D rendering engine 204 and virtual vantage point 306), or videodata packaging system 212 may generate different transports streams forthe color video data stream and depth video data stream associated witheach frame sequence 702.

Once a transport stream is generated, video data packaging system 212may provide the transport stream for streaming to a client-side mediaplayer device associated with a user. For example, video data packagingsystem 212 may stream (e.g., transmit by way of a network) the transportstream to the media player device itself, or may include the transportstream in a data pipeline in which the transport stream will be furtherprocessed and streamed to the media player device by another system(e.g., after being processed and/or repackaged by other devices,processes, and/or systems associated with the pipeline).

As mentioned above, in some examples, system 100 and/or other systems(e.g., other server-side systems) and devices described herein forrendering frames of a virtual scene from different vantage points basedon a virtual entity description frame may be used to generate virtualreality media content to be experienced by users. For example, inaddition to the operations described above, a virtual reality mediacontent provider system (e.g., within which system 100, video datapackaging system 212, and/or other devices and systems described hereinmay be included or with which these systems may otherwise be associated)may further generate and provide virtual reality media content based onthe transport stream that video data packaging system 212 generates andprovides. The virtual reality media content may be representative of avirtual scene and may be presentable to the user so as to be experiencedfrom a dynamically selectable virtual vantage point corresponding to anarbitrary virtual location with respect to the virtual scene. Forexample, the dynamically selectable virtual vantage point may beselected by the user of the media player device while the user isexperiencing the virtual scene using the media player device. Moreover,the virtual reality media content may be provided (e.g., by the virtualreality media content provider system that includes or is otherwiseassociated with system 100) to the media player device to allow the userto experience the virtual scene from the dynamically selectable virtualvantage point corresponding to the arbitrary virtual location within thevirtual scene.

To illustrate, FIG. 8 shows an exemplary configuration 800 in which anexemplary virtual reality media content provider system 802 (“providersystem 802”) that includes system 100 and video data packaging system212 generates virtual reality media content that is provided by way of anetwork 804 to an exemplary client-side media player device 806 (“mediaplayer device 806”) used by a user 808 to experience a virtual scene.

After one or more transport streams have been generated based on framesequences 702 as described above, provider system 802 may furtherencode, package, encrypt, or otherwise process the one or more transportstreams to form virtual reality media content that media player device806 may be configured to render. For example, the virtual reality mediacontent may include or be representative of a plurality of 2D video datastreams (e.g., 2D video data streams associated with color data and withdepth data for each virtual vantage point 306) that may be rendered bymedia player device 806 so as to present a view of virtual scene 300from any arbitrary virtual vantage point within virtual scene 300 (e.g.,including virtual vantage points other than virtual vantage points 306that may be of interest to user 808), as will be described below.Additionally or alternatively, the virtual reality media content mayinclude data representative of one or more volumetric models (e.g., 3Dor 4D models) of virtual objects included within virtual scene 300 thatalso may be rendered so as to be viewable from arbitrary virtual vantagepoints. The virtual reality media content may then be distributed by wayof network 804 to one or more media player devices such as media playerdevice 806 associated with user 808. For example, provider system 802may provide the virtual reality media content to media player device 806so that user 808 may experience virtual scene 300 virtually using mediaplayer device 806.

In some examples, it may be undesirable for user 808 to be limited toone or more discrete positions within an immersive virtual reality worldrepresented by the virtual reality media content (e.g., representativeof virtual scene 300). As such, provider system 802 may providesufficient data within the virtual reality media content representativeof virtual scene 300 to allow virtual scene 300 to be rendered not onlyfrom virtual vantage points 306, but from any dynamically selectablevirtual vantage point corresponding to an arbitrary virtual locationwithin virtual scene 300 (e.g. within or around virtual 3D space 302).For example, the dynamically selectable virtual vantage point may beselected by user 808 while user 808 is experiencing virtual scene 300using media player device 806.

As used herein, an “arbitrary virtual location” may refer to any virtualpoint in space associated with a virtual scene (e.g., within or around avirtual 3D space of the virtual scene). For example, arbitrary virtuallocations are not limited to fixed positions surrounding the virtualscene (e.g., fixed positions associated with virtual vantage points306), but also include all the positions between the positionsassociated with virtual vantage points 306 and positions inside ofvirtual 3D space 302. Moreover, arbitrary virtual locations may beassociated with arbitrary virtual vantage points not limited to aligningwith any of virtual vantage points 306. In some examples, such arbitraryvirtual locations may correspond to the most desirable virtual vantagepoints within virtual scene 300. For instance, if virtual scene 300includes a basketball game, user 808 may dynamically select virtualvantage points from which to experience the game that are in anyarbitrary virtual location on the basketball court. For example, theuser may dynamically select his or her virtual vantage points to followthe basketball up and down the basketball court and experience thebasketball game as if standing on the basketball court in the middle ofthe action of the game. In other words, for example, while virtualvantage points 306 may be positioned at fixed positions surrounding thebasketball court, user 808 may dynamically select arbitrary virtualvantage points from which to experience the game that are associatedwith any arbitrary position on the basketball court.

Network 804 may include a provider-specific wired or wireless network(e.g., a cable or satellite carrier network or a mobile telephonenetwork), the Internet, a wide area network, a content delivery network,or any other suitable network. Data may flow between provider system 802and media player device 806 (as well as other media player devices notexplicitly shown) using any communication technologies, devices, media,and protocols as may serve a particular implementation.

Media player device 806 may be used by user 808 to access and experiencevirtual reality media content received from provider system 802. Forexample, media player device 806 may be configured to generate (e.g.,based on the color video data stream and the depth video data stream foreach of the virtual vantage points included within the transport stream,which may be 2D video data streams) a 3D representation of virtual 3Dspace 302 of virtual scene 300 to be experienced by user 808 from anarbitrary virtual vantage point (e.g., a dynamically selectable virtualvantage point selected by the user and corresponding to an arbitraryvirtual location within virtual 3D space 302). To this end, media playerdevice 806 may include or be implemented by any device capable ofpresenting a field of view of an immersive virtual reality world (e.g.,an immersive virtual reality world representative of virtual scene 300)and detecting user input from user 808 to dynamically update theimmersive virtual reality world presented within the field of view asuser 808 experiences the immersive virtual reality world.

For example, FIG. 9 shows various exemplary types of media playerdevices 806 that may be used by user 808 to experience virtual realitymedia content. Specifically, as shown, media player device 806 may takeone of several different form factors such as a head-mounted virtualreality device 902 (e.g., a virtual reality gaming device) that includesa head-mounted display screen, a personal computer device 904 (e.g., adesktop computer, laptop computer, etc.), a mobile or wireless device906 (e.g., a smartphone, a tablet device, etc., possibly mounted to thehead of user 808 by means of a head mount apparatus), or by any otherdevice or configuration of devices that may serve a particularimplementation to facilitate receiving and/or presenting virtual realitymedia content. Different types of media player devices (e.g.,head-mounted virtual reality devices, personal computer devices, mobiledevices, etc.) may provide different types of virtual realityexperiences having different levels of immersiveness for user 808.

FIG. 10 illustrates an exemplary virtual reality experience 1000 inwhich user 808 is presented with exemplary virtual reality media contentrepresentative of a virtual scene as experienced from a dynamicallyselectable virtual vantage point corresponding to an exemplary arbitraryvirtual location with respect to the virtual scene. Specifically,virtual reality media content 1002 is presented within a field of view1004 that shows a virtual scene from a virtual vantage pointcorresponding to an arbitrary virtual location right underneath abasketball standard within the virtual 3D space of the virtual scenewhere a shot is being made. An immersive virtual reality world 1006based on the virtual scene may be available for the viewer to experienceby providing user input (e.g., head movements, keyboard input, etc.) tolook around and/or to move around (i.e., dynamically select a virtualvantage point from which to experience) immersive virtual reality world1006.

For example, field of view 1004 may provide a window through which user808 may easily and naturally look around immersive virtual reality world1006. Field of view 1004 may be presented by media player device 806(e.g., on a display screen of media player device 806) and may includevideo depicting objects surrounding the user within immersive virtualreality world 1006. Additionally, field of view 1004 may dynamicallychange in response to user input provided by user 808 as user 808experiences immersive virtual reality world 1006. For example, mediaplayer device 806 may detect user input (e.g., moving or turning thedisplay screen upon which field of view 1004 is presented). In response,field of view 1004 may display different objects and/or objects seenfrom a different virtual vantage point or virtual location in place ofthe objects seen from the previous virtual vantage point or virtuallocation.

In FIG. 10, immersive virtual reality world 1006 is illustrated as asemi-sphere, indicating that user 808 may look in any direction withinimmersive virtual reality world 1006 that is substantially forward,backward, left, right, and/or up from the virtual vantage point of thelocation under the basketball standard that user 808 has currentlyselected. In other examples, immersive virtual reality world 1006 mayinclude an entire 360° by 180° sphere such that user 808 may also lookdown. Additionally, user 808 may move around to other locations withinimmersive virtual reality world 1006 (i.e., dynamically selectingdifferent dynamically selectable virtual vantage points within thevirtual 3D space). For example, user 808 may select a virtual vantagepoint at half court, a virtual vantage point from the free-throw linefacing the basketball standard, a virtual vantage point suspended abovethe basketball standard, or the like.

FIG. 11 illustrates an exemplary method 1100 for rendering frames of avirtual scene from different vantage points based on a virtual entitydescription frame of the virtual scene. While FIG. 11 illustratesexemplary operations according to one embodiment, other embodiments mayomit, add to, reorder, and/or modify any of the operations shown in FIG.11. One or more of the operations shown in FIG. 11 may be performed bysystem 100 and/or by any implementation thereof.

In operation 1102, a virtual scene capture system may maintain datarepresentative of a plurality of virtual entities included within avirtual 3D space of a virtual scene. The plurality of virtual entitiesmay include a virtual object and a plurality of virtual vantage pointsinto the virtual 3D space. Specifically, for example, the plurality ofvirtual vantage points may include a first virtual vantage point and asecond virtual vantage point different from the first virtual vantagepoint. Operation 1102 may be performed in any of the ways describedherein.

In operation 1104, the virtual scene capture system may generate avirtual entity description frame representative of a state of at leastone virtual entity in the plurality of virtual entities at a particularpoint in a temporal sequence. For example, the virtual scene capturesystem may generate the virtual entity description frame based on themaintained data representative of the plurality of virtual entities.Operation 1104 may be performed in any of the ways described herein.

In operation 1106, the virtual scene capture system may provide thevirtual entity description frame generated in operation 1104 to aplurality of server-side 3D rendering engines associated with a contentprovider system. For example, the virtual scene capture system mayprovide the virtual entity description frame to a first 3D renderingengine associated with the first virtual vantage point and configured torender, based on the virtual entity description frame, a first surfacedata frame representative of color and depth data of surfaces of thevirtual object visible from the first virtual vantage point at theparticular point in the temporal sequence. Moreover, the virtual scenecapture system may provide the virtual entity description frame to asecond 3D rendering engine associated with the second virtual vantagepoint and configured to render, based on the virtual entity descriptionframe, a second surface data frame representative of color and depthdata of surfaces of the virtual object visible from the second virtualvantage point at the particular point in the temporal sequence.Operation 1106 may be performed in any of the ways described herein.

Various methods and systems have been described above for renderingframes of a virtual scene from different vantage points (e.g., virtualvantage points) based on virtual entity description frames of thevirtual scene. For example, as described above, system 100 may work inconjunction with other systems such as virtual scene control systems202, 3D rendering engines 204, asset storage system 208, and so forth,to maintain data representative of virtual entities included within avirtual 3D space of a virtual scene, generate virtual entity descriptionframes representative of the state of the virtual entities, and providethe virtual entity description frames to 3D rendering engines to rendersurface data frames representative of color and depth data.

Methods and systems for generating and providing virtual reality datathat accounts for level of detail will now be described. As will bedescribed and otherwise made apparent below, methods and systemsdescribed below for generating and providing virtual reality data thataccounts for level of detail may be associated with methods and systemsdescribed above in various ways. In some examples, as will be madeapparent below, methods and systems for generating and providing virtualreality data that accounts for level of detail may not be associatedwith methods and systems for rendering frames of a virtual scene fromdifferent vantage points based on virtual entity description frames, butrather may stand alone, or be implemented separately from such methodsand systems.

As used herein, virtual reality data may “account for level of detail”when different parts of a virtual 3D space are represented at differentlevels of quality or detail such as by using different pixel resolutions(e.g., different resolutions per unit distance such as different pixelsper meter, etc.), different numbers of bits to represent color or depth,or the like. In certain examples, methods and systems described hereinmay generate virtual reality data at select levels of detail based onthe amount of detail that a user may be expected to appreciate (e.g.,notice, distinguish, enjoy, etc.) when experiencing virtual realitycontent rendered based on the virtual reality data. For example, aparticular instance of virtual reality data may account for level ofdetail with respect to a particular vantage point if the instance ofvirtual reality data distinguishes between detail that will and will notbe appreciable from the particular vantage point. Specifically, thevirtual reality data may include a relatively large amount of dataassociated with a high level of detail (e.g., high pixel resolution perunit distance, high numbers of bits to represent rich colors and subtledifferences in depth, etc.) related to surfaces near the particularvantage point that a user experiencing the virtual 3D space from alocation at or near the particular vantage point may be able to fullyappreciate. At the same time, the virtual reality data may include asmaller amount of data associated with a lower level of detail relatedto surfaces that are relatively remote from the particular vantagepoint, and that the user may not be able to fully appreciate. Inaddition or as an alternative to accounting for level of detail, methodsand systems described below provide other benefits (e.g., related toefficiency, experience quality, etc.), as will be described and madeapparent below.

Methods and systems described below may leverage particular coordinatesystems, techniques, methodologies, etc., in order to generate andprovide virtual reality data that accounts for level of detail withrespect to particular vantage points of users experiencing the virtualreality data. In one implementation, for example, a virtual realityprovider system may access surface data (e.g., included within or linkedto by one or more surface data frames) representative of a virtual 3Dspace of a virtual scene. As with other surface data (e.g., surface dataframes) described herein, the accessed surface data may include bothcolor and depth data for surfaces included within the virtual 3D space.Based on the accessed surface data, the virtual reality provider systemmay orthographically project (e.g., render, construct, generate,reproject, etc.) a respective plurality of adjacent surface data slicesof the virtual 3D space along each of three orthogonal axes in acoordinate system associated with the virtual 3D space. For instance, aswill be described and illustrated in more detail below, the virtualreality provider system may orthographically project respectivepluralities of adjacent surface data slices (e.g., color data slices anddepth data slices) along each of a positive direction of an x-axis ofthe coordinate system, a negative direction of the x-axis of thecoordinate system, and positive and negative directions of both they-axis of the coordinate system and the z-axis of the coordinate system.Exemplary surface data slices will be described and illustrated in moredetail below.

Using the orthographically projected surface data slices, the virtualreality provider system may generate virtual reality data thatrepresents the virtual 3D space and accounts for level of detail of thesurfaces included within the virtual 3D space with respect to aparticular vantage point (e.g., a particular location within the virtual3D space from which the user may experience the virtual 3D space). Forexample, the virtual reality provider system may account for level ofdetail by representing, within the generated virtual reality data,certain surfaces (e.g., surfaces near the particular vantage point) at ahigher quality and/or level of detail represented by a relatively largeamount of data while representing other surfaces (e.g., surfaces moredistant from the particular vantage point) at a lower quality and/orlevel of detail represented by a relatively small amount of data. Toaccount for level of detail in this way, the virtual reality providersystem may, as part of the generating of the virtual reality data, applyone or more image transform operations (e.g., including cropping imagetransform operations, down-sampling image transform operations, layeringimage transform operations, etc.) to one or more surface data slices inthe respective pluralities of adjacent surface data slicesorthographically projected along the three orthogonal axes. Various suchimage transform operations that may be performed to account for level ofdetail in virtual reality data will be described in more detail below.

After the virtual reality data representing the virtual 3D space hasbeen generated, or while the virtual reality data is being generated(e.g., in real-time), the virtual reality provider system may providethe virtual reality data to a media player device associated with theparticular vantage point (e.g., a media player device that is presentingthe virtual 3D space from a perspective at or near the particularvantage point). For instance, the virtual reality provider system mayprovide the virtual reality data to the media player device forprocessing by the media player device to present (e.g., to a user of themedia player device) virtual reality content that is based on thevirtual 3D space and is tailored to the particular vantage point. Forexample, the virtual reality content may be tailored to the particularvantage point by including high quality (e.g., high resolution per unitdistance) representations of objects for which the user may be able toappreciate a large amount of detail (e.g., objects near the particularvantage point) and including low quality (e.g., low resolution per unitdistance) representations of objects for which the user may not be ableto appreciate significant detail (e.g., objects relatively distant fromthe particular vantage point).

Various benefits may arise from methods and systems described herein forgenerating and providing virtual reality data that accounts for level ofdetail. For example, by orthographically projecting respectivepluralities of adjacent surface data slices of the virtual 3D spacealong each of the three orthogonal axes of the coordinate system, avirtual reality provider system may systematically reduce or limit theamount of virtual reality data that is communicated to media playerdevices without degrading a perceived quality for users. For instance,image transform operations may be applied independently to each surfacedata slice to decrease the amount of data used to represent each surfacedata slice in accordance with, for example, how near each surface dataslice is to a particular vantage point or how much detail may beappreciated within each surface data slice from the perspective of theparticular vantage point. Thus, methods and systems described herein forgenerating and providing virtual reality data that accounts for level ofdetail may significantly decrease the amount of virtual reality datathat is transmitted to media player devices, and may correspondinglydecrease the amount of network bandwidth used to communicate such data.

This decrease in bandwidth usage and data transmission may benefitvirtual reality media providers, network providers, and users of mediaplayer devices receiving the virtual reality data. For example, networkcongestion may be reduced, data transmission times on the network may beshortened, power usage and cost of operating the network may decrease,battery life of media player devices may be increased, loading andbuffering wait times may be reduced or eliminated for virtual realitydata downloads, and so forth.

Moreover, the more efficient virtual reality data communication providedby methods and systems described herein may not correspond to anyappreciable decrease in quality experienced by users. This is becausethe only detail that may be removed or reduced from the virtual realitycontent is detail that users experiencing the virtual 3D space would notbe able to fully appreciate from their current vantage point anyway. Asthe users virtually move around from place to place within the virtual3D space (i.e., move from being located at or near one vantage point toanother), different instances of virtual reality data that are eachrepresentative of the virtual 3D space but are tailored to differentvantage points within the virtual 3D space may be dynamically providedto the users' media player devices such that appreciable detail isalways available to be presented to the users but bandwidth is notwasted communicating detail that the users may not be able toappreciate.

Along with decreasing bandwidth usage while maintaining a perceivedquality level for users, methods and systems for generating andproviding virtual reality data that accounts for level of detaildescribed herein may, in some respects, increase the quality of virtualreality experiences presented to users. For example, because a setnumber of bits (e.g., 8 bits in certain examples) may be dedicated torepresenting the depth of surfaces, depth data describing an entirevirtual 3D space may have relatively low depth resolution per unitdistance. However, by using the same number of bits to represent thedepth of surfaces in only a particular slice of the virtual 3D space (asopposed to the entire virtual 3D space), a higher depth resolution perunit distance may be achieved that may correspond to an increase inperceived quality to users.

Additionally, it will be understood that methods and systems describedherein may significantly conserve processing resources by reducing anamount of redundant 3D rendering that is performed. 2D image processingis generally less resource-intensive than 3D scene rendering because 2Dimage manipulation involves less computation than 3D rendering.Accordingly, by shifting detail customization work to image processorsand away from 3D renderers, systems and methods described herein mayprovide virtual reality data at higher levels of performance (e.g.,efficiency, accuracy, responsiveness, quality, etc.). Various additionaland/or alternative benefits that may be provided by methods and systemsfor generating and providing virtual reality data that accounts forlevel of detail will be made apparent in the description below.

FIG. 12 illustrates an exemplary virtual reality provider system 1200(“system 1200”) for generating and providing virtual reality data thataccounts for level of detail. In some examples, system 1200 may beassociated with system 100, described above. For instance, system 1200may be an exemplary implementation of system 100, system 100 may be anexemplary implementation of system 1200, one of systems 100 and 1200 mayinclude the other system (e.g., as a subsystem), systems 100 and 1200may interoperate with one another as separate systems, systems 100 and1200 may both be implemented within another system, or systems 100 and1200 may have another suitable association with one another. In otherexamples, systems 100 and 1200 may be separate systems that each operateindependently of the other. Regardless, systems 100 and 1200 may havevarious characteristics in common, and it will be understood thatexemplary implementations, configurations, definitions, and otherdisclosure provided above in relation to system 100 (and/orimplementations thereof) may similarly apply to system 1200 (andimplementations thereof), as will be described below.

System 1200 includes a plurality of facilities that interoperate togenerate and provide virtual reality data that accounts for level ofdetail. Specifically, as shown in FIG. 12, system 1200 may include,without limitation, a surface data access facility 1202, a surface dataprojection facility 1204, a virtual reality provisioning facility 1206,and a storage facility 1208 selectively and communicatively coupled toone another. It will be recognized that although facilities 1202 through1208 are shown to be separate facilities in FIG. 12, facilities 1202through 1208 may be combined into fewer facilities, such as into asingle facility, or divided into more facilities as may serve aparticular implementation. In some examples, each of facilities 1202through 1208 may be distributed between multiple devices (e.g., multipleservers or sets of servers communicatively coupled by way of networks)and/or multiple locations as may serve a particular implementation. Eachof facilities 1202 through 1208 will now be described in more detail.

Surface data access facility 1202 may include one or more physicalcomputing devices that may perform various operations. For example,surface data access facility 1202 may perform operations associated withaccessing surface data representative of a virtual 3D space of a virtualscene. The surface data accessed by surface data access facility 1202may include color and depth data for surfaces included within thevirtual 3D space. For instance, the surface data accessed by surfacedata access facility 1202 may be (or may be similar to) the color anddepth data described above in relation to surface data frames andsurface data frame sequences.

Surface data access facility 1202 may access the surface data in anysuitable manner. For example, surface data access facility 1202 mayaccess the surface data by capturing color, depth, and/or other data(e.g., audio data, metadata, etc.) for a real-world scene directly usingone or more capture devices included within system 1200, or may receive(e.g., request and download) such data from another system that hascaptured the data using such capture devices. In the same or otherexamples, surface data access facility 1202 may access surface data thatis based on a real-world scene or on a computer-generated scene byreceiving such data from an asset storage system (e.g., asset storagesystem 208) based on a virtual entity description frame such as one offrames 500. In other implementations, surface data access facility 1202may otherwise access any suitable surface data by any suitablemethodology described herein or as may serve a particularimplementation.

Surface data projection facility 1204 may include one or more physicalcomputing components (e.g., hardware and/or software components separatefrom or shared with those of surface data access facility 1202) thatperform various operations associated with orthographically projectingrespective pluralities of adjacent surface data slices of the virtual 3Dspace based on the surface data accessed by surface data access facility1202. For example, surface data projection facility 1204 mayorthographically project respective pluralities of adjacent surface dataslices along each of three orthogonal axes (e.g., including an x-axis, ay-axis, and a z-axis) in a coordinate system associated with the virtual3D space.

As used herein, an “orthographic projection” refers to a rendering orother projection of color data and/or depth data in which 3D surfaces of3D objects are projected onto a two-dimensional projection plane bymeans of a parallel projection in which projection lines are allorthogonal to the projection plane. Thus, an orthographic projection maycontrast with other renderings described herein (e.g., perspectiverenderings of virtual 3D space 302 performed by 3D rendering engines 204from perspectives of virtual vantage points 306). Specifically, whileperspective renderings may depict objects as the objects actually appearfrom a particular perspective by using projection lines that alloriginate and extend from a certain point (e.g., a particular vantagepoint) to the surfaces of objects, orthographic projections may depictobjects differently than the objects actually appear from any givenpoint in space by using parallel projection lines all orthogonal to aprojection plane, rather than extending from a common point.

Orthographic projections may be beneficial in various implementationsfor a variety of reasons. For example, as compared to perspectiveprojections, orthographic projections may have reduced overlap and,thus, reduced data redundancy. Additionally, orthographic projectionsmay facilitate a uniform segmentation of a virtual 3D space intorectangular cells, whereas frustum bounds associated with perspectiveprojections may make perspective projections more complicated and/ordifficult to align. Additionally, fewer orthographic projections may beused to uniformly sample a rectangular volume as compared to a number ofperspective projections used to uniformly sample the same volume.Exemplary perspective renderings of objects have been illustrated above(see, for example, depictions in surface data frames 210 in FIG. 6).Exemplary orthographic projections will be illustrated and described inmore detail below.

Virtual reality provisioning facility 1206 may include one or morephysical computing components (e.g., hardware and/or software componentsseparate from or shared with those of surface data access facility 1202and/or surface data projection facility 1204) that perform variousoperations associated with generating virtual reality data thatrepresents the virtual 3D space and accounts for level of detail of thesurfaces included within the virtual 3D space with respect to aparticular vantage point. For example, as will be described in moredetail below, the generating of the virtual reality data by virtualreality provisioning facility 1206 may include applying one or moreimage transform operations to one or more surface data slices in therespective pluralities of adjacent surface data slices orthographicallyprojected by surface data projection facility 1204.

Moreover, virtual reality provisioning facility 1206 may also provide,to a media player device associated with the particular vantage point,the virtual reality data that represents the virtual 3D space. Thevirtual reality data may be provided for processing by the media playerdevice to present (e.g., to a user of the media player device) virtualreality content that is based on the virtual 3D space and is tailored tothe particular vantage point. For example, as mentioned above, thevirtual reality content may be tailored to the particular vantage pointby having, for example, relatively high-quality depictions (e.g.,represented by relatively large amounts of data) of surfaces near theparticular vantage point in the virtual 3D space and relativelylow-quality depictions (e.g., represented by relatively small amounts ofdata) of surfaces remote from the particular vantage point in thevirtual 3D space.

Storage facility 1208 may maintain any suitable data received,generated, managed, tracked, maintained, used, and/or transmitted byfacilities 1202 through 1206 in a particular implementation. Forexample, as shown, storage facility 1208 may include surface data 1210,which may include color and/or depth data (e.g., included in surfacedata frames, surface data frame sequences, and/or in any other format)representative of one or more virtual 3D spaces of one or more virtualscenes (e.g., virtual scenes based on real-world scenes,computer-generated scenes, a mixture of both, etc.). Storage facility1208 may further include virtual reality data 1212, which may includeone or more instances of virtual reality data configured to providevirtual reality content associated with one or more different vantagepoints. For instance, for a particular virtual 3D space of a particularvirtual scene, virtual reality data 1212 may include a plurality ofdifferent instances of virtual reality data that are associated withdifferent vantage points at different locations within the virtual 3Dspace. Storage facility 1208 may also maintain additional or alternativedata as may serve a particular implementation.

FIG. 13 illustrates an exemplary configuration 1300 in which anexemplary implementation of system 1200 generates and provides virtualreality data accounting for level of detail for exemplary media playerdevices. Specifically, as shown, configuration 1300 illustrates variousserver-side components, some or all of which may implement system 1200,as well as client-side media player devices associated with respectiveusers. The server-side components include a surface data access facility1302, at least one network 1304 (e.g., networks 1304-1 and 1304-2), aplurality of orthographic projection engines 1306 (e.g., orthographicprojection engines 1306-1 through 1306-K), and a plurality of detailcustomization engines 1308 (e.g., detail customization engines 1308-1through 1308-M). Another network 1310 is shown to separate theserver-side components of configuration 1300 (e.g., the implementationof system 1200) from a plurality of client-side media player devices1312 (e.g., media player devices 1312-1 through 1312-N). Each of mediaplayer devices 1312 is associated with a respective user 1314 (e.g.,users 1314-1 through 1314-N). Each of the entities in configuration 1300will now be described.

Surface data access facility 1302 may include or be implemented by oneor more computing systems and may be configured to perform the same orsimilar operations as surface data access facility 1202, describedabove. For example, surface data access facility 1302 may access surfacedata by capturing real-world surface data (e.g., color and/or depth datarepresentative of surfaces of real-world objects in a real-world scene)using capture devices that are included within or otherwise controlledby or accessed by surface data access facility 1302. In certainimplementations, surface data access facility 1302 may access data froman implementation of system 100 (not explicitly shown) that, asdescribed above, may maintain virtual entities (e.g., in accordance withphysics-based object behaviors, Al-based object behaviors, commands fromone or more virtual scene control systems to modify maintained data,etc.) and generate virtual entity description frames representative of astate of the virtual entities. Accordingly, based on such virtual entitydescription frames, surface data access facility 1302 may access surfacedata from a database such as asset storage system 208 based on links tothe surface data included in the virtual entity description frames. Inthe same or other examples, surface data access facility 1302 mayreceive, capture, load, and/or otherwise access surface data in any ofthe ways described herein or as may serve a particular implementation.

Networks 1304 and 1310 may be implemented by or include any of the typesof networks or network technologies described herein. For example,networks 1304 and 1310 may be part of or may include provider-specificwired or wireless networks (e.g., a cable or satellite carrier networkor a mobile telephone network), the Internet, wide area networks,content delivery networks, or any other suitable networks. As such, datamay flow over networks 1304 and/or 1310 using any communicationtechnologies, devices, media, and protocols as may serve a particularimplementation. In certain examples, networks 1304 (i.e., networks1304-1 and 1304-2) may be local and/or private networks exclusivelymanaged and/or used by a virtual reality media provider or the like,while network 1310 may include one or more public networks (e.g., apublic carrier network, the Internet, etc.) to allow data to betransmitted from server-side systems to client-side media playerdevices. While networks 1304-1, 1304-2 and 1310 are each illustrated asseparate networks, it will be understood that some or all of thesenetworks may be implemented as a single network, or that subcomponentsof the networks may be common among two or more of the networks.

In some configurations, one or more of networks 1304 or 1310 may beomitted (e.g., replaced by direct connections, non-network interfaceconnections, etc.). For example, in a configuration where surface dataaccess facility 1302 is implemented on the same computing resources asorthographic projection engines 1306, network 1304-1 may be omitted assurface data access facility 1302 may be integrated with orthographicprojection engines 1306 and/or may communicate accessed surface data toorthographic projection engines 1306 directly (e.g., without using anetwork). Similarly, in the same or other examples, some or all oforthographic projection engines 1306 and detail customization engines1308 may be implemented on common computing resources, reducing oreliminating the use of network 1304-2 for communicating datarepresentative of surface data slices. In still other examples, networks1304 and/or 1310 may be divided into more networks than shown inconfiguration 1300 if that should serve a particular implementation.

Collectively, orthographic projection engines 1306 may implement or beincluded within surface data projection facility 1204 described above.As such, orthographic projection engines 1306 may be configured toperform the same or similar operations described above in relation tosurface data projection facility 1204. In order to generate all theorthographic projections of the different pluralities of adjacentsurface data slices of the virtual 3D space, in certain examples, adifferent individual orthographic projection engine 1306 may be used togenerate each orthographic projection of each data slice in eachplurality of data slices. For example, different orthographic projectionengines 1306 may be used to generate the color data slice and the depthdata slice of each surface data slice, or orthographic projectionengines 1306 may each individually generate entire surface data slicesincluding color and depth data slices. For instance, in someimplementations, one orthographic projection engine 1306 (e.g.,orthographic projection engine 1306-1) may generate a first color dataslice along an x-axis of a coordinate system associated with the virtual3D space, another orthographic projection engine 1306 (e.g.,orthographic projection engine 1306-2) may generate a first depth dataslice along the x-axis of the coordinate system (e.g., corresponding tothe first color data slice), and so forth for all of the color dataslices and depth data slices in all of the respective pluralities ofadjacent surface data slices of the virtual 3D space along theorthogonal axes in the coordinate system associated with the virtual 3Dspace. In other implementations, one orthographic projection engine 1306(e.g., orthographic projection engine 1306-1) may generate a firstsurface data slice along the x-axis of a coordinate system associatedwith the virtual 3D space (e.g., including both a color data slice and adepth data slice), another orthographic projection engine 1306 (e.g.,orthographic projection engine 1306-2) may generate a second surfacedata slice along the x-axis of the coordinate system, and so forth forall of the surface data slices in all of the respective pluralities ofadjacent surface data slices of the virtual 3D space along theorthogonal axes in the coordinate system associated with the virtual 3Dspace.

In certain implementations, each orthographic projection engine 1306 maybe implemented using separate, independent computing resources (e.g.,implemented on separate server computers, on separate processors of asingle computer, etc.). In other examples, one or more of orthographicprojection engines 1306 may be combined so as to run on common (e.g.,shared) computing resources. For instance, multiple orthographicprojection engines 1306 (e.g., up to and including all of orthographicprojection engines 1306) may operate on a same server computer, on asame processor, or the like. In these examples, orthographic projectionengines 1306 sharing computer hardware resources may be configured torun as separate processes or threads in software. As shown, an integer,K, of orthographic projection engines 1306 may be utilized toorthographically project all of the pluralities of surface data slicesused in a particular implementation.

In some examples, orthographic projection engines 1306 may implement orbe implemented by 3D rendering engines 204 described above. In contrast,in implementations of system 1200 that are not closely related withsystem 100, orthographic projection engines 1306 at least may beanalogous to 3D rendering engines 204 in the sense that both 3Drendering engines 204 and orthographic projection engines 1306 areconfigured to operate to generate (e.g., render, project, etc.)respective renderings or projections of certain portions of a virtual 3Dspace from different perspectives or projection planes within thevirtual 3D space. However, whereas 3D rendering engines 204 may each beconfigured to render perspective depictions of the virtual 3D space(e.g., renderings as the space may appear from a particular virtualvantage point within the virtual 3D space), orthographic projectionengines 1306 may orthographically project pluralities of adjacentsurface data slices (e.g., orthographic projections from particularprojection planes), as will be described and illustrated in more detailbelow.

Each of orthographic projection engines 1306 may transmit, by way ofnetwork 1304-2, a particular data slice (e.g., color data slice, depthdata slice, full surface data slice, etc.) that the orthographicprojection engine 1306 is responsible for orthographically projecting toeach of detail customization engines 1308. Accordingly, in someexamples, detail customization engines 1308-1 through 1308-M may receiveor otherwise gain access to every surface data slice generated by all ofthe orthographic projection engines 1306-1 through 1306-K (e.g., all ofthe surface data slices in all of the respective pluralities of adjacentsurface data slices of the virtual 3D space along each of the threeorthogonal axes in the coordinate system). In other examples, detailcustomization engines 1308 may be in communication with orthographicprojection engines 1306 (e.g., by way of network 1304-2) to dynamicallycommunicate to orthographic projection engines 1306 which surface dataslices the detail customization engines 1308 are currently using toprepare virtual reality datasets for each particular vantage point. Assuch, further efficiency in system 1200 may be achieved as detailcustomization engines 1308 may not receive or process surface dataslices not actually being used to generate virtual reality data for aparticular vantage point, and as certain orthographic projection engines1306 may abstain (e.g., at least temporarily) from generating theirassigned orthographic projections (e.g., in cases where the assignedorthographic projections are not currently being used by any detailcustomization engines 1308).

Collectively, detail customization engines 1308 may be analogous tovideo data packaging system 212 in that detail customization engines1308 may receive projections or renderings of surface data and process,package, transmit, and/or otherwise perform operations to generate anddistribute virtual reality data to media player devices. Moreparticularly, the plurality of detail customization engines 1308 mayimplement virtual reality provisioning facility 1204 described above togenerate and provide different instances of virtual reality data thataccount for level of detail of surfaces included within the virtual 3Dspace with respect to different vantage points (e.g., different vantagepoints associated with different media player devices 1312).

As with orthographic projection engines 1306, detail customizationengines 1308 may be implemented on independent computing resources orshared computing resources. In some examples, each detail customizationengine 1308 may be associated with a different respective vantage pointwithin the virtual 3D space of the virtual scene. As such, each detailcustomization engine 1308 may be configured to: 1) access the respectivepluralities of adjacent surface data slices generated by orthographicprojection engines 1306; 2) generate a respective instance of virtualreality data that represents the virtual 3D space and accounts for levelof detail of the surfaces included within the virtual 3D space withrespect to the respective vantage point with which each detailcustomization engine 1308 is associated (e.g., including by applying oneor more image transform operations to one or more of the accessedsurface data slices); and 3) provide the respective instance of virtualreality data to any of media player devices 1312 that may be associatedwith the respective vantage point for processing by the media playerdevices 1312 to present (e.g., to a respective user 1314) virtualreality content that is based on the virtual 3D space and is tailored tothe respective vantage point.

In some examples, detail customization engines 1308 may concurrentlyperform these operations (e.g., including the providing of therespective instances of virtual reality data for processing by thedifferent media player devices 1312). Accordingly, media player devices1312 may be able to concurrently request and receive, from appropriatedetail customization engines 1308, appropriate instances of virtualreality data associated with particular vantage points that respectiveusers 1314 dynamically select as they experience different parts of thevirtual 3D space of the virtual scene. Various image transformoperations that may be performed by detail customization engines 1308 togenerate virtual reality data that accounts for level of detail withrespect to different vantage points will be described and illustrated inmore detail below.

Media player devices 1312 may be implemented by any suitable systems ordevices such as any of the media player devices described herein thatare configured to receive virtual reality data (e.g., from one or moreof detail customization engines 1308 by way of network 1310) and torender the virtual reality data to present virtual reality content torespective users 1314. At any given point in time, each media playerdevice 1312 may be associated with a particular vantage point for whichat least one of detail customization engines 1308 is generating virtualreality data (e.g., virtual reality data that accounts for level ofdetail with respect to that particular vantage point). However, at thegiven point in time, different media player devices 1312 may beassociated with different particular vantage points and, as such, may bereceiving different instances of virtual reality data from differentdetail customization engine 1308. Moreover, as users 1314 experience thevirtual 3D space of the virtual scene and move about in the virtual 3Dspace, respective media player devices 1312 may switch from beingassociated with certain vantage points to being associated with othervantage points. Accordingly, a particular media player device 1312 mayswitch from receiving one instance of virtual reality data from onedetail customization engine 1308 to receiving another instance ofvirtual reality data from another detail customization engine 1308.

To clearly illustrate how system 1200 may generate and provide virtualreality data that accounts for level of detail in a configuration suchas configuration 1300, various aspects of the methods and systems forgenerating and providing virtual reality data that accounts for level ofdetail described above will now be illustrated and described in moredetail.

FIG. 14 illustrates an exemplary virtual 3D space 1400 of a virtualscene based upon which system 1200 may generate and provide virtualreality data that accounts for level of detail according to principlesdescribed herein. As shown, virtual 3D space 1400 includes variousexemplary objects such as a sofa object 1402, a chair object 1404, and alamp object 1406 in a living room setting. While the living room settingof virtual 3D space 1400 may be relatively small and may include arelatively small number of objects, it will be understood that otherexamples of virtual 3D spaces may include any type of space or scenethat may be presented for users to experience in virtual reality and, assuch, may be larger or smaller than virtual 3D space 1400, may containmore or fewer objects than virtual 3D space 1400, and so forth. In someexamples, a particular virtual scene (e.g., a scene representative of abuilding or a large area, etc.) may include a plurality of differentvirtual 3D spaces (e.g., rooms of the building, different sections ofthe large area, etc.).

As described above, as used herein, virtual 3D spaces and virtual scenesmay be “virtual” in the sense that a user may experience them in avirtual manner (e.g., using virtual reality technology such as a virtualreality media player device) rather than experiencing them in real life(e.g., without use of such virtual reality technology). However, whilevirtual 3D spaces of virtual scenes such as virtual 3D space 1400 may beexperienced virtually, it will be understood that objects within thevirtual 3D spaces and the virtual 3D spaces themselves may be generatedin any suitable manner to be based on real-world, virtualized (e.g.,computer-generated), and/or other types of objects and/or scenery.

For example, virtual 3D scene 1400 may be a virtual representation of areal-world scene and at least some of the color and depth data includedin the surface data from which virtual 3D space 1400 is generated may becaptured by a plurality of capture devices disposed at differentpositions with respect to the real-world scene. In other words, theliving room setting depicted in virtual 3D space 1400, as well asobjects 1402 through 1406, may exist in the real world, and virtual 3Dspace 1400 may be generated based on data captured by one or morecapture devices disposed in the real-world living room setting tocapture surface data representative of the real-world objects.

In other examples, at least some of the color and depth data included inthe surface data from which virtual 3D space 1400 is generated may be 3Dmodel data representative of a computer-generated virtual object. Forexample, certain aspects of virtual 3D space 1400 (e.g., one or more ofobjects 1402 through 1406) or the entirety of virtual 3D space 1400 maybe virtualized (e.g., computer-generated) based on 3D models rather thanbased on real-world objects and/or scenery. Virtual scenes that includeboth camera-captured real-world objects and/or scenery along withvirtualized objects and/or scenery may be referred to as mixed realityvirtual scenes.

In examples where virtual 3D space 1400 is based on real-world objectsand/or scenery, system 1200 may be configured to access the surface datarepresentative of a real-world scene (e.g., surface data captured by aplurality of capture devices disposed around the real-world scene) andprocess the data in accordance with methods and systems described hereinin real time. As used herein, operations may be performed in “real time”when they are performed immediately and without undue delay such that,for example, data processing operations associated with an ongoing event(e.g., a real-world sporting event, concert, etc.) are performed whilethe event is still ongoing (e.g., rather than after the fact) even ifthere is some amount of delay such as a few seconds or minutes. In someexamples, these types of real-time operations may allow virtual realityusers to experience a real-world event live or at approximately the sametime as people attending the event are experiencing it.

As further illustrated in FIG. 14, virtual 3D space 1400 may beassociated with a coordinate system 1408. Coordinate system 1408 may beany suitable type of coordinate system, such as a cartesian coordinatesystem, a polar coordinate system, or the like. In the example of FIG.14, coordinate system 1408 is illustrated as a cartesian coordinatesystem that includes an x-axis stretching in a +x and a −x direction, ay-axis stretching in a +y and a −y direction, and a z-axis stretching ina +z and a −z direction. As shown, each of the coordinate axes x, y, andz are orthogonal to one another in 3D space.

As described above, respective pluralities of adjacent surface dataslices of virtual 3D spaces such as virtual 3D space 1400 may begenerated along each of three orthogonal axes in a coordinate systemsuch as the x, y, and z axes of coordinate system 1408. To illustrate,FIGS. 15A through 15F show exemplary pluralities of adjacent surfacedata slices of virtual 3D space 1400. While all three axes of coordinatesystem 1408 are not explicitly shown in FIGS. 15A through 15F, it willbe understood that each of FIGS.15A through 15F are depicted using acommon orientation so that coordinate system 1408 remains consistentlyoriented in each of the figures. As shown, relevant information relatedto coordinate system 1408 (e.g., relevant indications of coordinate axisdirections for +x, −x, +y, −y, +z, or −z directions) are indicated ineach of FIGS. 15A through 15F.

As used herein, a “data slice” may refer to an orthographic projectionof color data, depth data, or a combination of both color and depth datafor surfaces within a particular sub-volume (e.g., a relatively thin,“slice”-shaped rectangular prism or another suitably shaped sub-volume)of the entire volume of a virtual 3D space. Such orthographicprojections of the sub-volume may be referred to herein as “color dataslices” when they include color data, as “depth data slices” when theyinclude depth data, and as “surface data slices” when they include bothcolor and depth data. Thus, in certain examples, a surface data slicemay include or refer to (e.g., as an abstraction for simplicity ofdescription) both a color data slice and a depth data slice. Forinstance, it will be understood that the surface data slices in FIGS.15A through 15F may each actually represent both a color data slice anda depth data slice, so that these figures could alternatively be drawnto include twice as many slices (i.e., color and depth data slices foreach direction of each axis in coordinate system 1408).

The surface data slices illustrated in FIGS. 15A through 15F willcollectively be referred to herein as surface data slices 1502, but itis noted that each surface data slice may be individually referred to bya numbering notation that will now be described. As shown, each surfacedata slice 1502 is labeled with a “1502” as well as with a sign (i.e., a“+” or a “−”) followed by an axis designation (i.e., “x”, “y”, or “z” inthese examples), followed by an index number (i.e., “1”, “2”, or “3” inthese examples). Based on the sign, axis designation, and index number,each surface data slice may be uniquely identified as being projected ina direction (indicated by the sign) along an axis (indicated by the axisdesignation) as a particular index (indicated by the index number) ofother slices in a plurality of slices being projected in the samedirection along the same axis.

For example, a surface data slice 1502+x1 refers to a first surface dataslice in a plurality of surface data slices in the positive (“+”)direction along the x-axis. Similarly, a surface data slice 1502+x2refers to a second surface data slice in the same plurality of surfacedata slices as surface data slice 1502+x1. As another example, a surfacedata slice 1502−x1 refers to a first surface data slice in a pluralityof surface data slices that are also along the x-axis but are projectedin the negative (“−”) direction.

Accordingly, as shown in FIG. 15A, the orthographically projectedpluralities of adjacent surface data slices 1502 of virtual 3D space1400 may include a plurality of adjacent surface data slices 1502+x(e.g., surface data slices 1502+x1 through 1502+x3) orthographicallyprojected in a positive direction along the x-axis of coordinate system1408.

Similarly, as shown in FIG. 15B, the orthographically projectedpluralities of adjacent surface data slices 1502 of virtual 3D space1400 may include a plurality of adjacent surface data slices 1502−x(e.g., surface data slices 1502−x1 through 1502−x3) orthographicallyprojected in a negative direction along the x-axis of coordinate system1408.

As shown in FIG. 15C, the orthographically projected pluralities ofadjacent surface data slices 1502 of virtual 3D space 1400 may include aplurality of adjacent surface data slices 1502+y (e.g., surface dataslices 1502+y1 through 1502+y3) orthographically projected in a positivedirection along the y-axis of coordinate system 1408.

Similarly, as shown in FIG. 15D, the orthographically projectedpluralities of adjacent surface data slices 1502 of virtual 3D space1400 may include a plurality of adjacent surface data slices 1502−y(e.g., surface data slices 1502−y1 through 1502−y3) orthographicallyprojected in a negative direction along the y-axis of coordinate system1408.

As shown in FIG. 15E, the orthographically projected pluralities ofadjacent surface data slices 1502 of virtual 3D space 1400 may include aplurality of adjacent surface data slices 1502+z (e.g., surface dataslices 1502+z1 through 1502+z3) orthographically projected in a positivedirection along the z-axis of coordinate system 1408.

Similarly, as shown in FIG. 15F, the orthographically projectedpluralities of adjacent surface data slices 1502 of virtual 3D space1400 may include a plurality of adjacent surface data slices 1502−z(e.g., surface data slices 1502−z1 through 1502−z3) orthographicallyprojected in a negative direction along the z-axis of coordinate system1408.

In the example of FIGS. 15A through 15F, each plurality of surface dataslices 1502 is shown to include three surface data slices 1502, which,as explained above, may represent three respective color data slices andthree respective depth data slices (not explicitly drawn ordifferentiated in FIGS. 15A through 15F). However, it will be understoodthat a given virtual 3D space may be sliced in any way and/or into anynumber of surface data slices as may serve a particular implementation.For instance, in some implementations, a different number of data slices(i.e., other than three) may be included in each plurality of surfacedata slices. Additionally, different numbers of data slices may beincluded in different pluralities (e.g., there may be more or fewer dataslices along an x-axis than along a y-axis of the coordinate system, forexample).

Certain advantages may arise when a relatively large number ofrelatively thin surface data slices are used for a particular pluralityof surface data slices (e.g., in a particular direction along aparticular axis). For example, system 1200 may have more flexibilitywhen processing larger numbers of thinner surface data slices togenerate virtual reality data by applying different image transformoperations to different surface data slices. As such, virtual realitydata may be more closely tailored to particular vantage points and highdetail of both pixel and depth resolution per unit distance may beachieved to provide a high-quality experience to a particular user 1314.However, less data may be used to represent relatively few and/orrelatively thick surface data slices. Accordingly, an optimum number ofsurface data slices may be determined for each virtual 3D spaceaccording to various design goals and/or technical limitations of eachparticular implementation.

As such determinations are made as to how a particular virtual 3D spaceis to be sliced and how many slices are to be made in a particulardirection along a particular axis, it will be understood that anysuitable number of surface data slices (e.g., including numbers otherthan three) may be included in each plurality of surface data slices.Moreover, it will be understood that certain numbers of surface dataslices may be inherently beneficial for certain data structures that maybe employed in particular implementations. For example, if eachplurality of surface data slices in each of FIGS. 15A through 15F wereto use a number of surface data slices 1502 that is a power of two(e.g., two slices per plurality, four slices per plurality, eight slicesper plurality, sixteen slices per plurality, or the like, rather thanthree slices per plurality), octree-type data structures couldconveniently be used to store, organize, analyze, facilitate selectionof and/or otherwise process surface data slices 1502 with respect tocoordinate system 1408.

While FIGS. 15A through 15F illustrate which sub-volume of virtual 3Dspace 1400 is associated with each surface data slice 1502, as well aswhich direction along which axis each surface data slice 1502 isorthographically projected, FIGS. 15A through 15F do not explicitly showthe color or depth orthographic projections for each surface data slice1502. As such, FIGS. 16A through 16C illustrate exemplary orthographicprojection views (shown below in FIGS. 16A through 16C) corresponding tosub-volume views (shown above in FIGS. 16A through 16C) for a fewparticular surface data slices 1502.

Specifically, FIG. 16A illustrates an orthographic projection view forsurface data slice 1502+x1, FIG. 16B illustrates an orthographicprojection view for surface data slice 1502+y1, and FIG. 16C illustratesan orthographic projection view for surface data slice 1502−z1. Each ofthe orthographic projection views for the surface data slices 1502 shownin FIGS. 16A through 16C may depict, for example, a color orthographicprojection or a depth orthographic projection for the respective surfacedata slice 1502 as viewed from an orthographic perspective indicated bya respective arrow extending from the orthographic projection view tothe sub-volume view of the surface data slice 1502.

As shown in FIG. 16A, the orthographic projection view of surface dataslice 1502+x1 includes an orthographic projection of one end of sofaobject 1402 that is included in surface data slice 1502+x1. It will beunderstood that if any other objects were to be included within surfacedata slice 1502+x1, those objects would also be included within theorthographic projection. Additionally, it will be understood that theorthographic projection view of surface data slice 1502+x1 illustratedmay represent only color data or depth data, and that a correspondingorthographic projection corresponding to the other type of data may alsobe associated with (e.g., included as a part of) surface data slice1502+x1.

Similarly, as shown in FIG. 16B, the orthographic projection view ofsurface data slice 1502+y1 includes an orthographic projection of oneend of chair object 1404 that is included in surface data slice 1502+y1,while, as shown in FIG. 16C, the orthographic projection view of surfacedata slice 1502−z1 includes an orthographic projection of the top oflamp object 1406 that is included in surface data slice 1502−z1. It willbe understood that if any other objects were to be included,respectively, within surface data slices 1502+y1 and/or 1502−z1, thoseobjects would also be included within the respective orthographicprojection. Additionally, it will be understood that the respectiveorthographic projection views of surface data slices 1502+y1 and 1502−z1illustrated may represent only color data or depth data, and thatrespective orthographic projections corresponding to the other type ofdata may also be associated with (e.g., included as a part of) surfacedata slices 1502+y1 and/or 1502−z1.

While not explicitly illustrated in the exemplary orthographicprojection views of FIGS. 16A through 16C, it will be understood thatcertain surface data slices 1502 may slice through objects. For example,while surface data slice 1502+x1 may include an end of sofa object 1402for which surface data is available, surface data slice 1502+x2 mayslice through sofa object 1402. The surface data (e.g., color data ordepth data) for an object that is sliced into by a surface data slice1502 may be represented or depicted within an orthographic projection inany manner as may serve a particular implementation. For instance, insome implementations, the surface data may be omitted from theorthographic projection so that the object being sliced through isignored or treated as being invisible for the surface data slice thatcuts through it. In other implementations, the surface data may beincluded in the orthographic projection, but may appear to be reversed(e.g., inside out) since surfaces of the object are essentially beingprojected from the inside of the object rather than from the outside.

In still other implementations, the surface data may be replaced by anull value such as a value for black, white, or another constant coloror depth value. These implementations may also provide or facilitatefurther efficiency optimizations as a result of empty or null sectionsof surface data slices being identified and culled out such that dataprocessing will only be performed on pixels having actual data (i.e.,rather than null values) and less data in output virtual realitydatasets is wasted on null value data. Consequently, in addition tosurface data slices being cropped or dropped based on their proximityfrom the particular vantage point, these implementations may allowsurface data slices to be cropped, dropped, or otherwise reducedentirely based on the data (or lack thereof) that they contain. Fordynamic scenes, such changes may occur during runtime such thatadjustments to particular image transform operations and/or to a sliceordering within a slice atlas data structure (described in more detailbelow) may be made during runtime.

In certain implementations, a combination of any or all of these ways ofprocessing the surface data or other suitable ways of processing thesurface data for sliced-through objects may be employed as may serve aparticular implementation.

Once system 1200 (e.g., orthographic projection engines 1306 implementedas part of surface data projection facility 1204) has generated eachsurface data slice 1502 in the pluralities of adjacent surface dataslices 1502 of virtual 3D space 1400 along the orthogonal axes incoordinate system 1408, the surface data slices 1502 may be communicatedfor use in generating virtual reality data (e.g., by each detailcustomization engine 1308). For example, as described above, the dataincluded in each surface data slice 1502 may be transmitted or otherwisecommunicated from one entity to another (e.g., from orthographicprojection engines 1306 to each of detail customization engines 1308).For example, surface data slices 1502 may be communicated from onecomputer or system of computers to another (e.g., by way of a networksuch as network 1304-2), communicated from one software program orprocess running on a computer to another, communicated from one locationin computer memory to another, or communicated in any other suitable waybetween two suitable entities as may serve a particular implementation.

In certain implementations, system 1200 may package surface data slices1502 together into a single data structure for transmission to one ormore other entities. For example, based on the output of eachorthographic projection engine 1306, system 1200 may arrange eachsurface data slice 1502 in the pluralities of adjacent surface dataslices to form a slice atlas data structure representative of surfacedata slices 1502 in the pluralities of adjacent surface data slices.Specifically, for example, contemporaneous frames (i.e., framesrepresentative of virtual 3D space 1400 at the same or a similar momentin time) from surface data slices 1502 may be arranged onto a single“mega-frame” of the slice atlas data structure. Consequently, thegenerating of the virtual reality data (e.g., by detail customizationengines 1308) may be performed using the slice atlas data structurerepresentative of surface data slices 1502 in the pluralities ofadjacent surface data slices.

FIG. 17 illustrates an exemplary slice atlas data structure 1700 formedby an arrangement of surface data slices 1502 in the pluralities ofadjacent surface data slices 1502. For example, as shown in FIG. 17,slice atlas data structure 1700 includes a plurality of frames 1702(e.g., frames 1702-1 through 1702-P, where P is an integer of how manyframes are included in a particular sequence) that each include anarrangement of contemporaneous frames of surface data slices 1502.Frames 1702 may also be referred to herein as “mega-frames” todistinguish the mega-frames from the individual frames of each surfacedata slice 1502.

Formatting and/or arranging surface data slices 1502 for transmission orcommunication to other entities using slice atlas data structure 1700may be beneficial at least because each frame 1702 may include all ofthe orthographic projection information that a particular detailcustomization engine 1308 may use to generate a single frame of virtualreality data. Accordingly, in certain examples, a data packaging enginenot explicitly shown in FIG. 13 may receive output surface data slicesfrom each orthographic projection engine 1306 and may package thesurface data slices to form mega-frames of a slice atlas data structuresuch as slice atlas data structure 1700 for transmission to one or moreof detail customization engines 1308. However, it will be understoodthat, in other examples, such packaging may not be performed and eachsurface data slice may be communicated to a respective detailcustomization engine 1308 in any other suitable way.

Slice atlas data structure 1700 illustrates an arrangement that includeseach of the eighteen surface data slices 1502 illustrated in FIGS. 15Athrough 15F. Since eighteen equally-sized surface data slices 1502 arepacked together onto a 5×4 mega-frame grid, two spots within the gridare not used to hold any particular surface data slice 1502. As shown,these unused spots within the grid may hold a null-value (e.g., blackpixels) or may otherwise be used to hold data (e.g., meta-data)associated with the surface data slices 1502 included within slice atlasdata structure 1700. It will be understood that surface data slices 1502may be packed together in any order and/or in any arrangement as mayserve a particular implementation. For example, in certainimplementations, different surface data slices 1502 may be differentsizes and thus may be packed together in other arrangements that may beefficient or convenient for these particular implementations.

It will be understood that one slice atlas data structure 1700 may beused for color orthographic projections of surface data slices 1502,while a separate slice atlas data structure 1700 may be used for depthorthographic projections of surface data slices 1502. Alternatively, inother examples, an implementation of slice atlas data structure 1700 maybe used that includes both color and depth orthographic projections ofsurface data slices 1502 on a single slice atlas data structure. Forinstance, such an implementation of slice atlas data structure 1700 maybe approximately twice the size of the implementation illustrated inFIG. 17 to hold at least thirty-six surface data slices (i.e., eighteencolor data slices and eighteen depth data slices).

In order to indicate to a receiving entity (e.g., an entity that willreceive slice atlas data structure 1700 when it is communicated) whatdata is included within slice atlas data structure 1700, slice atlasdata structure 1700 may include metadata descriptive of the surface dataslices 1502 arranged in slice atlas data structure 1700. As such, thegenerating of virtual reality data by the receiving entity (e.g., aparticular detail customization engine 1308) may be performed usingslice atlas data structure 1700 in accordance with the metadata. Forexample, metadata included within or otherwise associated with sliceatlas data structure 1700 may indicate, for each surface data slice 1502included within each frame 1702, camera transform information, a slicedirection indicator, a coordinate axis indicator, an index indicator,mega-frame coordinates demarcating boundaries of the surface data slice1502 within the frame 1702, and/or any other metadata as may serve aparticular implementation.

As mentioned above, it may be desirable to tailor different instances ofvirtual reality data representative of the same virtual 3D space withdifferent levels of detail based on different vantage points associatedwith the different instances of virtual reality data. For example,virtual reality data to be used by a media player device presenting thevirtual 3D space from the perspective of a vantage point in one locationin the virtual 3D space may include different details about differentobjects than virtual reality data to be used by another media playerdevice presenting the virtual 3D space from the perspective of a vantagepoint in a different location at the other side of the virtual 3D space.

To illustrate, FIG. 18 shows a plurality of exemplary vantage pointsdisposed at different locations within virtual 3D space 1400.Specifically, a vantage point 1802 may be disposed near one corner ofvirtual 3D space 1400 near one end of sofa object 1402 but relativelyfar from, for example, chair object 1404 and lamp object 1406. As such,it may be desirable for virtual reality data used to present virtual 3Dspace 1400 from the perspective of vantage point 1802 to includesignificant detail descriptive of sofa object 1402, while less detailmay be included for chair object 1404 and/or lamp 1406 because, fromvantage point 1802, that detail may be less appreciable to a user.

Conversely, as further shown, a vantage point 1804 may be disposed nearanother corner of virtual 3D space 1400 near the other end of sofaobject 1402 and also near one end of chair object 1404 and lamp object1406. As such, it may be desirable for virtual reality data used topresent virtual 3D space 1400 from the perspective of vantage point 1804to include more detail descriptive of chair object 1404 and lamp object1406 than was included in the virtual reality data used to present thevirtual 3D space from the perspective of vantage point 1802. However,while a significant amount of detail may be appreciable of the end ofsofa object 1402 nearer to vantage point 1804, less detail may beappreciable from vantage point 1804 of the other end of sofa object 1402and, as such, some of the detail included in the virtual reality dataassociated with vantage point 1802 may be omitted from virtual realitydata associated with vantage point 1804.

Other vantage points 1806 and 1808 illustrate other specific locationswithin virtual 3D space 1400 for which still other combinations oflevels of detail for different objects may be appropriate. For example,a relatively high level of detail for chair object 1404 and relativelylow levels of detail for lamp object 1406 and/or sofa object 1402 may beappropriate for vantage point 1806, while relatively low levels ofdetail for all of the objects in virtual 3D space 1400 may beappropriate for the perspective of vantage point 1808, which is notparticularly nearby any of the objects.

In certain implementations, vantage points such as vantage points 1802through 1808 may be arbitrarily and/or dynamically positioned atparticular locations of interest, at locations from which users indicatethey would like to experience the virtual 3D space, and/or at othersuitable locations selected in any manner as may serve a particularimplementation. In other implementations, however, vantage points may bedistributed evenly and/or in accordance with a regular pattern in orderto divide up the virtual 3D space in a consistent and regular fashion.

To illustrate an example, FIG. 19 shows a plurality of exemplary vantagepoints 1900 disposed at different locations within virtual 3D space1400. Specifically, in contrast to the example of FIG. 18 where a fewvantage points were placed at arbitrary locations within virtual 3Dspace 1400, vantage points 1900 (i.e., each illustrated by ‘X’ symbols,some of which are explicitly designated as vantage points 1900 but allof which are to be considered vantage points 1900) in FIG. 19 are placedat locations in accordance with a regular pattern (e.g., a grid-likepattern) so that each surface data slice includes a three-by-three gridof nine vantage points 1900. As such, vantage points 1900 are evenlydistributed throughout virtual 3D space 1400 in an organized manner.

By evenly distributing the twenty-seven vantage points 1900 throughoutvirtual 3D space 1400 as shown in FIG. 19, virtual 3D space 1400 mayeffectively be divided into a set of twenty-seven distinct sub-volumes.As such, in this particular example, twenty-seven different detailcustomization engines 1308 may be used to generate twenty-sevendifferent instances of virtual reality data accounting for level ofdetail of surfaces included within virtual 3D space 1400 in differentways for the twenty-seven different vantage points 1900. Media playerdevices 1312 may thus be given a choice of up to twenty-seven differentversions of the virtual reality data and may select whichever instanceof the virtual reality data is most appropriate at a particular time(e.g., based on a current vantage point that the respective user 1314desires). In other examples, other numbers and distributions of vantagepoints 1900 within virtual 3D space 1400 may be used.

As mentioned above, the number of slices into which virtual 3D space1400 is divided may make certain data structures (e.g., octrees, etc.)convenient to use. For a similar reason, certain data structures may beconvenient for use with certain distribution patterns of vantage points1900. For example, if the pattern of vantage points 1900 is a cubic gridwith a power of two on each side (e.g., a 2×2×2 grid, a 4×4×4 grid, an8×8×8 grid, etc.), an octree data structure may be used to efficientlyorganize data associated with each vantage point 1900.

For example, a recursive octree-based algorithm may be used to determinewhich instance of virtual reality data to send to a particular mediaplayer device associated with a particular vantage point. Specifically,an octree cell may be recursively split if the vantage point with whichthe media player device is associated is inside the cell. At each layerof recursion, if the vantage point is associated with a single virtualreality data instance (the base case of the recursive algorithm), thatvirtual reality data instance may be provided to the media playerdevice. Otherwise, the octree cell may continue to be split (e.g., intoits eight child nodes, each of which is associated with at least one ofits own virtual reality data instances) until reaching the base casewhere the single virtual reality data instance may be provided.

In other examples, more flexible methodologies, includingnon-octree-based algorithms and/or algorithms similarly based on othertypes of data structures, may be used instead of octree-basedalgorithms. For example, when a particular virtual 3D space is firstconfigured, a manual or automated process (e.g., a script or the like)may be used to generate a data structure that each detail customizationengine may read in to use as a “recipe” for generating virtual realitydata associated with a particular vantage point. The process may loopover each coordinate axis and handle each surface data slice inaccordance with the proximity of the particular vantage point to eachsurface data slice or any other suitable factor. Based on thisproximity, for instance, the process may prescribe certain imagetransform operations that are to be performed for each surface dataslice, and may perform other suitable operations as may serve aparticular implementation (e.g., slice packing operations to arrange thedata slices into a slice atlas data structure or the like).

As has been described, detail customization engines 1308 may apply oneor more image transform operations to one or more surface data slices1502 in order to generate virtual reality data that represents thevirtual 3D space and accounts for level of detail of surfaces within thevirtual 3D space with respect to a particular vantage point such as oneof vantage points 1900. Such image transform operations may include anysuitable operations that may affect what data is to be included, withrespect to a particular surface data slice, within an instance ofvirtual reality data. For instance, in some examples, an image transformoperation may include removing a particular surface data slice frombeing included within the virtual reality data associated with theparticular vantage point, or may include adding the particular surfacedata slice to the virtual reality data without modifying the particularsurface data slice. In other examples, an image transform operationapplied to the particular surface data slice may include a croppingimage transform operation, a down-sampling image transform operation, alayering image transform operation, and/or any other suitable imagetransform operation as may serve a particular implementation.

To illustrate, FIGS. 20A through 20C illustrate exemplary imagetransform operations 2000 (e.g., image transform operations 2000-1through 2000-3) that may be performed on surface data slices 1502 ofvirtual 3D space 1400. Each image transform operation 2000 will now bedescribed.

FIG. 20A illustrates a down-sampling image transform operation 2000-1 inwhich a quality level of a surface data slice 1502-1 (which mayrepresent any of surface data slices 1502 described above) is reduced sothat a surface data slice 2002-1 is represented using less data than anamount of data used to represent surface data slice 1502-1 at anoriginal quality level. Down-sampling image transform operation 2000-1may be performed in any suitable way to remove certain detail fromsurface data slice 2002-1 to thereby reduce the amount of data used torepresent surface data slice 2002-1 as compared to representing surfacedata slice 1502-1. For example, various aspects of quality or detail maybe reduced including pixel resolution per unit distance, frame rate,color palette, depth resolution per unit distance, and/or any otheraspect of quality or detail as may serve a particular implementation.

FIG. 20B illustrates a cropping image transform operation 2000-2 inwhich two areas of surface data slice 1502-2 (which may represent any ofsurface data slices 1502 described above) are differentiated so as to beprocessed separately in a surface data slice 2002-2. Specifically, asshown, surface data slice 2002-2 includes a “higher quality area” in thebottom left corner that retains that high quality of 1502-2, while therest of surface data slice 2002-2 (e.g., the area that has been croppedout) has been down-sampled to a lower quality that is represented usingless data. For example, the lower quality area may be down-sampled orprocessed in any of the ways described herein, such as in one of theways that surface data slice 2002-1 is down-sampled.

FIG. 20C illustrates a layering image transform operation 2000-3 inwhich a surface data slice 1502-3A (which may represent any of surfacedata slices 1502 described above) is combined with another surface dataslice 1502-3B (which also may represent any of surface data slices 1502described above that is separate from surface data slice 1502-3A) suchthat surface data slices 1502-3A and 1502-3B are represented in acombined surface data slice 2002-3 that is represented using less datathan an amount of data used to represent surface data slices 1502-3A and1502-3B separately. For example, two or more adjacent surface dataslices 1502 that are relatively distant from a vantage point for whichvirtual reality data is being generated may be combined if they containfew details appreciable from the vantage point, contain similar detailsas one another as far as may be appreciated from the perspective of thevantage point, or the like.

Image transform operations such as illustrated by image transformoperations 2000 and/or other suitable image transform operations may beapplied to surface data slices in any way and in any combination so asto generate virtual reality data that accounts for level of detail withrespect to a particular vantage point. For example, system 1200 (e.g., aparticular detail customization engine 1308 generating virtual realitydata that accounts for level of detail with respect to the particularvantage point) may generate the virtual reality data by determining atarget level of detail for a particular surface data slice based on aproximity of the particular surface data slice to the particular vantagepoint. Based on the target level of detail, system 1200 may determinethat the surface data slice is to be transformed to account for level ofdetail with respect to the particular vantage point. Then, based on thedetermination that the surface data slice is to be transformed toaccount for level of detail with respect to the particular vantagepoint, system 1200 may select the image transform operation to beapplied from a plurality of available image transform operations (e.g.,including image transform operations such as image transform operations2000 or other image transform operations described herein).

To illustrate a specific example, FIG. 21 shows how an exemplaryinstance of virtual reality data may be generated to represent virtual3D space 1400 and to account for level of detail of surfaces includedwithin virtual 3D space 1400 with respect to a particular vantage point2100. In FIG. 21, each of surface data slices 1502 (i.e., pluralities ofsurface data slices 1502+x, 1502−x, 1502+y, 1502−y, 1502+z, and 1502−z)are illustrated and labeled around the edges of virtual 3D space 1400with respect to coordinate system 1408.

As shown, vantage point 2100 is located in a corner of virtual 3D space1400 that is included within surface data slices 1502+x1, 1502−x3,1502+y3, 1502−y1, 1502+z3, and 1502−z1. As such, in order to provide asmuch detail as possible for surfaces near vantage point 2100, each ofthese surface data slices may be included without a reduction in detailwithin the virtual reality data instance associated with vantage point2100. Alternatively, a cropping image transform operation may be appliedto one or more of these surface data slices in order to include, withinthe virtual reality data, a relatively large level of detail for aportion of the surface data slice near vantage point 2100 and a smallerlevel of detail for areas further away from vantage point 2100.

Immediately adjacent to the surface data slices in which vantage point2100 is located are surface data slices 1502+x2, 1502−x2, 1502+y2,1502−y2, 1502+z2, and 1502−z2. Because much of the detail includedwithin these surface data slices may be appreciable from vantage point2100, each of these surface data slices may be included within thevirtual reality data after applying, for example, a relatively minordown-sampling image transform operation to reduce the amount of dataused to represent these surface data slices. In some examples, only partof these surface data slice may be down-sampled (e.g., using acombination of a cropping image transform operation and a down-samplingimage transform operation), or these surface data slices may be layeredtogether with one or more of the other surface data slices that are evenmore distant from vantage point 2100, as described below.

More remote from vantage point 2100 (e.g., not adjacent to the surfacedata slices in which vantage point 2100 is located) are surface dataslices 1502+x3, 1502−x1, 1502+y1, 1502−y3, 1502+z1, and 1502−z3. Becausemuch of the detail included within these surface data slices may not beappreciable from vantage point 2100, each of these surface data slicesmay be included within the virtual reality data after applying, forexample, a relatively significant down-sampling image transformoperation to reduce the amount of data used to represent these surfacedata slices. In some examples, only part of these surface data slice maybe down-sampled (e.g., using a combination of a cropping image transformoperation and a down-sampling image transform operation), or one or moreof these surface data slices may be omitted from the virtual realitydata altogether. As mentioned above, these surface data slices may also,in some implementations, be layered together with one or more of theother surface data slices that are adjacent to vantage point 2100 usinglayering image transform operations. For example, while differentdetails may be appreciable from vantage point 2100 in surface dataslices 1502+x2 and 1502+x3 such that it may be appropriate to keep thesesurface data slice separate, details included within surface data slices1502−x1 and 1502−x2 may not be appreciably different from one anotherfrom vantage point 2100, thus making a layering image transformoperation of these two surface data slices appropriate.

While various image transform operations have been described above forspecific surface data slices 1502 with respect to vantage point 2100, itwill be understood that these image transform operations are exemplaryonly, and that any suitable image transform operations may be applied toany surface data slices 1502 as may serve a particular implementation.Additionally, it will be understood that, while the focus of the exampleshown in FIG. 21 was on discrete image transform operations beingapplied to particular surface data slices 1502, any combination of imagetransform operations may be applied to one or more surface data slices1502 as may serve a particular implementation.

FIG. 22 illustrates an exemplary method for generating and providingvirtual reality data that accounts for level of detail. While FIG. 22illustrates exemplary operations according to one embodiment, otherembodiments may omit, add to, reorder, and/or modify any of theoperations shown in FIG. 22. One or more of the operations shown in FIG.22 may be performed by system 1200 and/or by any implementation thereof.

In operation 2202, a virtual reality provider system may access surfacedata representative of a virtual 3D space of a virtual scene. Forexample, the surface data may include color and depth data for surfacesincluded within the virtual 3D space. Operation 2202 may be performed inany of the ways described herein.

In operation 2204, the virtual reality provider system mayorthographically project a respective plurality of adjacent surface dataslices of the virtual 3D space along each of three orthogonal axes in acoordinate system associated with the virtual 3D space. For example, thevirtual reality provider system may orthographically project thepluralities of adjacent surface data slices based on the surface dataaccessed in operation 2202. Operation 2204 may be performed in any ofthe ways described herein.

In operation 2206, the virtual reality provider system may generatevirtual reality data that represents the virtual 3D space and accountsfor level of detail of the surfaces included within the virtual 3D spacewith respect to a particular vantage point. In some examples, operation2206 may include applying an image transform operation to a surface dataslice in one of the respective pluralities of adjacent surface dataslices orthographically projected in operation 2204. Operation 2206 maybe performed in any of the ways described herein.

In operation 2208, the virtual reality provider system may provide thevirtual reality data that represents the virtual 3D space and that wasgenerated in operation 2206 to a media player device associated with theparticular vantage point. For example, the virtual reality providersystem may provide the virtual reality data for processing by the mediaplayer device to present, to a user of the media player device, virtualreality content that is based on the virtual 3D space and is tailored tothe particular vantage point. Operation 2208 may be performed in any ofthe ways described herein.

In certain embodiments, one or more of the systems, components, and/orprocesses described herein may be implemented and/or performed by one ormore appropriately configured computing devices. To this end, one ormore of the systems and/or components described above may include or beimplemented by any computer hardware and/or computer-implementedinstructions (e.g., software) embodied on at least one non-transitorycomputer-readable medium configured to perform one or more of theprocesses described herein. In particular, system components may beimplemented on one physical computing device or may be implemented onmore than one physical computing device. Accordingly, system componentsmay include any number of computing devices, and may employ any of anumber of computer operating systems.

In certain embodiments, one or more of the processes described hereinmay be implemented at least in part as instructions embodied in anon-transitory computer-readable medium and executable by one or morecomputing devices. In general, a processor (e.g., a microprocessor)receives instructions, from a non-transitory computer-readable medium,(e.g., a memory, etc.), and executes those instructions, therebyperforming one or more processes, including one or more of the processesdescribed herein. Such instructions may be stored and/or transmittedusing any of a variety of known computer-readable media.

A computer-readable medium (also referred to as a processor-readablemedium) includes any non-transitory medium that participates inproviding data (e.g., instructions) that may be read by a computer(e.g., by a processor of a computer). Such a medium may take many forms,including, but not limited to, non-volatile media, and/or volatilemedia. Non-volatile media may include, for example, optical or magneticdisks and other persistent memory. Volatile media may include, forexample, dynamic random access memory (“DRAM”), which typicallyconstitutes a main memory. Common forms of computer-readable mediainclude, for example, a disk, hard disk, magnetic tape, any othermagnetic medium, a compact disc read-only memory (“CD-ROM”), a digitalvideo disc (“DVD”), any other optical medium, random access memory(“RAM”), programmable read-only memory (“PROM”), electrically erasableprogrammable read-only memory (“EPROM”), FLASH-EEPROM, any other memorychip or cartridge, or any other tangible medium from which a computercan read.

FIG. 23 illustrates an exemplary computing device 2300 that may bespecifically configured to perform one or more of the processesdescribed herein. As shown in FIG. 23, computing device 2300 may includea communication interface 2302, a processor 2304, a storage device 2306,and an input/output (“I/O”) module 2308 communicatively connected via acommunication infrastructure 2310. While an exemplary computing device2300 is shown in FIG. 23, the components illustrated in FIG. 23 are notintended to be limiting. Additional or alternative components may beused in other embodiments. Components of computing device 2300 shown inFIG. 23 will now be described in additional detail.

Communication interface 2302 may be configured to communicate with oneor more computing devices. Examples of communication interface 2302include, without limitation, a wired network interface (such as anetwork interface card), a wireless network interface (such as awireless network interface card), a modem, an audio/video connection,and any other suitable interface.

Processor 2304 generally represents any type or form of processing unit(e.g., a central processing unit and/or a graphics processing unit)capable of processing data or interpreting, executing, and/or directingexecution of one or more of the instructions, processes, and/oroperations described herein. Processor 2304 may direct execution ofoperations in accordance with one or more applications 2312 or othercomputer-executable instructions such as may be stored in storage device2306 or another computer-readable medium.

Storage device 2306 may include one or more data storage media, devices,or configurations and may employ any type, form, and combination of datastorage media and/or device. For example, storage device 2306 mayinclude, but is not limited to, a hard drive, network drive, flashdrive, magnetic disc, optical disc, RAM, dynamic RAM, other non-volatileand/or volatile data storage units, or a combination or sub-combinationthereof. Electronic data, including data described herein, may betemporarily and/or permanently stored in storage device 2306. Forexample, data representative of one or more executable applications 2312configured to direct processor 2304 to perform any of the operationsdescribed herein may be stored within storage device 2306. In someexamples, data may be arranged in one or more databases residing withinstorage device 2306.

I/O module 2308 may include one or more I/O modules configured toreceive user input and provide user output. One or more I/O modules maybe used to receive input for a single virtual reality experience. I/Omodule 2308 may include any hardware, firmware, software, or combinationthereof supportive of input and output capabilities. For example, I/Omodule 2308 may include hardware and/or software for capturing userinput, including, but not limited to, a keyboard or keypad, atouchscreen component (e.g., touchscreen display), a receiver (e.g., anRF or infrared receiver), motion sensors, and/or one or more inputbuttons.

I/O module 2308 may include one or more devices for presenting output toa user, including, but not limited to, a graphics engine, a display(e.g., a display screen), one or more output drivers (e.g., displaydrivers), one or more audio speakers, and one or more audio drivers. Incertain embodiments, I/O module 2308 is configured to provide graphicaldata to a display for presentation to a user. The graphical data may berepresentative of one or more graphical user interfaces and/or any othergraphical content as may serve a particular implementation.

In some examples, any of the facilities described herein may beimplemented by or within one or more components of computing device2300. For example, one or more applications 2312 residing within storagedevice 2306 may be configured to direct processor 2304 to perform one ormore operations or functions associated with facilities 102 or 104 ofsystem 100 (see FIG. 1) and/or facilities 1202, 1204, or 1206 of system1200 (see FIG. 12). Likewise, either or both of storage facilities 106(of system 100) and 1208 (of system 1200) may be implemented by orwithin storage device 2306.

To the extent the aforementioned embodiments collect, store, and/oremploy personal information provided by individuals, it should beunderstood that such information shall be used in accordance with allapplicable laws concerning protection of personal information.Additionally, the collection, storage, and use of such information maybe subject to consent of the individual to such activity, for example,through well known “opt-in” or “opt-out” processes as may be appropriatefor the situation and type of information. Storage and use of personalinformation may be in an appropriately secure manner reflective of thetype of information, for example, through various encryption andanonymization techniques for particularly sensitive information.

In the preceding description, various exemplary embodiments have beendescribed with reference to the accompanying drawings. It will, however,be evident that various modifications and changes may be made thereto,and additional embodiments may be implemented, without departing fromthe scope of the invention as set forth in the claims that follow. Forexample, certain features of one embodiment described herein may becombined with or substituted for features of another embodimentdescribed herein. The description and drawings are accordingly to beregarded in an illustrative rather than a restrictive sense.

What is claimed is:
 1. A method comprising: generating, by a virtualreality provider system based on surface data representative of at leastone of color characteristics or depth characteristics of surfaces withina three-dimensional (3D) space, a surface data projection of a portionof the 3D space; and applying, by the virtual reality provider system,an image transform operation to the surface data projection to transformthe surface data projection to account for a level of detail of thesurfaces within the 3D space with respect to a particular vantage pointwithin the 3D space.
 2. The method of claim 1, further comprising:determining, by the virtual reality provider system, a target level ofdetail for the surface data projection based on a proximity of thesurface data projection to the particular vantage point; anddetermining, by the virtual reality provider system based on the targetlevel of detail, that the surface data projection is to be transformedto account for the level of detail with respect to the particularvantage point; wherein the applying of the image transform operation tothe surface data projection is performed based on the determining thatthe surface data projection is to be transformed to account for thelevel of detail with respect to the particular vantage point.
 3. Themethod of claim 2, further comprising selecting, by the virtual realityprovider system from a plurality of available image transformoperations, the image transform operation that is applied to the surfacedata projection to transform the surface data projection to account forthe level of detail with respect to the particular vantage point;wherein the selecting is performed based on the determination that thesurface data projection is to be transformed and based on the targetlevel of detail.
 4. The method of claim 3, wherein the selected imagetransform operation that is applied to the surface data projectioncomprises a cropping image transform operation in which two areas of thesurface data projection are differentiated so as to be processedseparately.
 5. The method of claim 3, wherein the selected imagetransform operation that is applied to the surface data projectioncomprises a down-sampling image transform operation in which a qualitylevel of the surface data projection is reduced so that the surface dataprojection is represented using less data than an amount of data used torepresent the surface data projection at an original quality level. 6.The method of claim 3, wherein the selected image transform operationthat is applied to the surface data projection comprises a layeringimage transform operation in which the surface data projection iscombined with an additional surface data projection such that thesurface data projection and the additional surface data projection arerepresented using less data than an amount of data used to represent thesurface data projection and the additional surface data projectionseparately.
 7. The method of claim 3, wherein the selected imagetransform operation that is applied to the surface data projectioncomprises a combination of a cropping image transform operation and adown-sampling image transform operation in which a first area and asecond area of the surface data projection are differentiated and aquality level of only the first area is reduced so that the first areais represented using less data than an amount of data used to representthe second area.
 8. The method of claim 1, wherein the surface dataprojection comprises a surface data slice included in a plurality ofadjacent surface data slices of the 3D space arranged along an axis in acoordinate system associated with the 3D space.
 9. The method of claim8, further comprising: determining, by the virtual reality providersystem, that the surface data slice is to be transformed to account forthe level of detail with respect to the particular vantage point,wherein the applying of the image transform operation to the surfacedata projection is performed based on the determining that the surfacedata slice is to be transformed; determining, by the virtual realityprovider system, that an additional surface data slice of the pluralityof adjacent surface data slices is not to be transformed to account forthe level of detail with respect to the particular vantage point; andabstaining from applying, by the virtual reality provider system inresponse to the determining that the additional surface data slice isnot to be transformed, any image transform operation to the additionalsurface data projection.
 10. The method of claim 1, wherein the surfacedata projection is implemented as an orthographic projection of the 3Dspace or a perspective projection of the 3D space.
 11. The method ofclaim 1, wherein the surface data includes both: color datarepresentative of the color characteristics of the surfaces within the3D space; and depth data representative of the depth characteristics ofthe surfaces within the 3D space.
 12. The method of claim 1, furthercomprising: generating, by the virtual reality provider system based onthe surface data projection to which the image transform operation isapplied, virtual reality data that represents the 3D space and accountsfor the level of detail of the surfaces within the 3D space with respectto the particular vantage point; and providing, by the virtual realityprovider system to a media player device associated with the particularvantage point, the virtual reality data for processing by the mediaplayer device to present, to a user of the media player device, virtualreality content that is based on the 3D space and is tailored to theparticular vantage point.
 13. A system comprising: a memory storinginstructions; and a processor communicatively coupled to the memory andconfigured to execute the instructions to: generate, based on surfacedata representative of at least one of color characteristics or depthcharacteristics of surfaces within a three-dimensional (3D) space, asurface data projection of a portion of the 3D space; and apply an imagetransform operation to the surface data projection to transform thesurface data projection to account for a level of detail of the surfaceswithin the 3D space with respect to a particular vantage point withinthe 3D space.
 14. The system of claim 13, wherein the processor isfurther configured to execute the instructions to: determine a targetlevel of detail for the surface data projection based on a proximity ofthe surface data projection to the particular vantage point; anddetermine, based on the target level of detail, that the surface dataprojection is to be transformed to account for the level of detail withrespect to the particular vantage point; and wherein the applying of theimage transform operation to the surface data projection is performedbased on the determining that the surface data projection is to betransformed to account for the level of detail with respect to theparticular vantage point.
 15. The system of claim 14, wherein theprocessor is further configured to execute the instructions to select,from a plurality of available image transform operations, the imagetransform operation that is applied to the surface data projection totransform the surface data projection to account for the level of detailwith respect to the particular vantage point; wherein the selecting isperformed based on the determination that the surface data projection isto be transformed and based on the target level of detail.
 16. Thesystem of claim 15, wherein the selected image transform operation thatis applied to the surface data projection comprises one of: a croppingimage transform operation in which two areas of the surface dataprojection are differentiated so as to be processed separately; adown-sampling image transform operation in which a quality level of thesurface data projection is reduced so that the surface data projectionis represented using less data than an amount of data used to representthe surface data projection at an original quality level; and a layeringimage transform operation in which the surface data projection iscombined with an additional surface data projection such that thesurface data projection and the additional surface data projection arerepresented using less data than an amount of data used to represent thesurface data projection and the additional surface data projectionseparately.
 17. The system of claim 15, wherein the selected imagetransform operation that is applied to the surface data projectioncomprises a combination of a cropping image transform operation and adown-sampling image transform operation in which a first area and asecond area of the surface data projection are differentiated and aquality level of only the first area is reduced so that the first areais represented using less data than an amount of data used to representthe second area.
 18. The system of claim 13, wherein the surface dataprojection comprises a surface data slice included in a plurality ofadjacent surface data slices of the 3D space arranged along an axis in acoordinate system associated with the 3D space, and the processor isfurther configured to execute the instructions to: determine that thesurface data slice is to be transformed to account for the level ofdetail with respect to the particular vantage point, wherein theapplying of the image transform operation is performed based on thedetermining that the surface data slice is to be transformed; determinethat an additional surface data slice of the plurality of adjacentsurface data slices is not to be transformed to account for the level ofdetail with respect to the particular vantage point; and abstain fromapplying, by the virtual reality provider system in response to thedetermining that the additional surface data slice is not to betransformed, any image transform operation to the additional surfacedata projection.
 19. The system of claim 13, wherein the processor isfurther configured to execute the instructions to: generate, based onthe surface data projection to which the image transform operation isapplied, virtual reality data that represents the 3D space and accountsfor the level of detail of the surfaces within the 3D space with respectto the particular vantage point; and provide, to a media player deviceassociated with the particular vantage point, the virtual reality datafor processing by the media player device to present, to a user of themedia player device, virtual reality content that is based on the 3Dspace and is tailored to the particular vantage point.
 20. Anon-transitory computer-readable medium storing instructions that, whenexecuted, direct a processor of a computing device to: generate, basedon surface data representative of at least one of color characteristicsor depth characteristics of surfaces within a three-dimensional (3D)space, a surface data projection of a portion of the 3D space; and applyan image transform operation to the surface data projection to transformthe surface data projection to account for a level of detail of thesurfaces within the 3D space with respect to a particular vantage pointwithin the 3D space.